How to write an exchange with 50 suppliers and not go crazy

Suppliers are different. Some are ready to adapt to our format - others are not; Some exchange SOAP'om - others REST'om; some work with product codes - others with offer IDs; some are ready to give the status on order - others are not; some have reference books, the elements of which you need to compare with your own - others do not. In general, very different.
How to write an exchange with 50 suppliers and not go crazy

Data exchange with suppliers needs to be automated. 21st century in the yard. It is clear that in an ideal world it is necessary to convene a conference of suppliers and consumers in this industry (in my case these are auto parts), on which to agree a common format for information exchange and everyone will be happy. The pharmacists have this. But we live in a non-ideal world.
The original idea of ​​all is one: the supplier is ready to give you the elements of their directories and is ready to take your order from you. But for all suppliers, business processes have their own characteristics and this imposes different initial requirements on API vendors, and different programmers implementing them bring the picture to the point that the APIs of different suppliers have no common features, except for the original idea.
How to write an exchange in fifty formats? Write fifty exchanges? Let's assume that at best, exchange with one provider to write and test for 1 week and a programmer's rate of 3000 r /h. Multiply? And I did not even remember about the support and administration of this zoo.
There are other options? My answer: write one exchange!
We must come up with a 51st supplier and write an exchange with him. This should be ideal for us, for our processes supplier. This supplier should have the similarities of all suppliers, differing from them in detail.

We should start thinking not "from the supplier", but "from us" and the picture will instantly be simplified. At us that business process one and the programmer too one and everything that goes beyond the reasonable in our opinion - we are not particularly interesting. It is clear that such an approach will inevitably lead to "harassment" in logic and some "fine" functionality will go under the knife, but this is the price of simplicity.
I counted three such types of loadings:
The functionality we need, but which some suppliers will not be ready to provide
The functionality that suppliers provide, but we do not need it.
Functionality realized by cardinally different methods from different suppliers.
When I came up with my ideal supplier, I shoveled the vendors' documentation to their APIs and developed 4 main methods by which I developed the API of my ideal vendor.

Method 1: Generalization of

Generalization is the basic principle of developing any functionality. Exchange with suppliers is no exception.
For example, one supplier can use the statuses of orders "accepted", "in work", "shipped", "denied". Another "in the work of the manager", "in work in the warehouse", "in the work of the transport company", "closed successfully", "canceled". Another supplier I saw 27 different statuses. Seriously, I specially counted them. It is clear that in our system all this zoo we do not need. We summarize the statuses to the minimum required set: "new", "in the supplier's work", "shipped by the supplier", "problem".
Next, you just need to compare each element of the directory of each vendor with the element of our directory.

Method 2: Separation of

You can not generalize everything. Sometimes it is necessary to separate.
For example, if one supplier accepts the entire order (all order lines in one XML) and the other according to the shopping cart principle (add the goods to the basket - add the goods to the basket - order a basket), then write something in a "general" way it will turn out.
In this case, the exchange with our ideal 51st supplier should still be divided. those. he must optionally be able to send the order by one and the other method. At me it is called "method of sending of the order" and it has two variants of value: "the order entirely" and "line by line".
And still there is a setting "method of checking the status" and it already has three options: "request for order - return on order", "query for order - return line by line" and "query line by line - return line by line".
But to get involved in division is not worth much, because each crushing leads to a complication of the system. And what exactly you should not do is allow tree-like separation. Try to reduce everything to flat settings. If the fragmentation of functionality according to one characteristic will depend on the fragmentation in another way, you are buried in the settings and instead of elegant simplicity you will get the hell of the settings.

Method 3: Clipping

The clipping method is intuitive and is used in cases where the supplier's functionality is superior to the one you need.
For example, a vendor can provide the ability to edit an order until a certain point (for example, before being sent to the set).
But we are not particularly interested in this and all other suppliers do not have such an opportunity, so we just need to say that we will not use this opportunity.
It is necessary to clearly identify the boundaries of automation. Everything that goes beyond them should be excluded from consideration. Ignore the arguments "this additional convenience": if you can do without the other suppliers without this, then this is also possible.

Method 4: Default

The default method is the inverse of the clipping method. In this case, the problem is that the automation boundary includes functionality that the provider does not support. For example, the return of the order status. Some suppliers may not support this.
In this case, it is not worth checking in all subsequent algorithms for checking whether the status of the order is relevant or ignoring it, because the supplier does not know how to return it. It is much easier to put a "stub", which will translate such orders into the last successful status.
Thus, by analyzing your needs and capabilities - you can draw the boundaries of automation. This will be the end of the API design phase of your ideal 51st supplier. Ie, if you go into the technique - the suppliers in the system will still be 5? but the upload /download code will be the same and will generate XML for all 50 suppliers in the 51st vendor format.
It so happened that in this article I focused on downloading the order, as the most variable and complex part of the exchange (at least for me), but everything written can be applied to both downloading and uploading and ordering and master data and something else, anything else.
Next, you need to consider how best to arrange the upload file for the ideal 51st supplier:
First, it should be XML (why - it will become clear later).
Secondly, it should contain the maximum amount of data that you can unload with this order. Those. at this stage it does not matter to us whether it is required to specify the VAT rate for the supplier to whom the order is ultimately intended - we must unload everything that may be required.
Thirdly, the converted values ​​must be converted. Those. already at the stage of forming XML for the ideal 51st supplier, for example, product identifiers should be substituted from the supplier to which the order is intended. This should not cause any difficulties, since all the data for this in the order is.
Thus, we should have a file containing all the data necessary for the supplier to accept the order, but in a slightly invalid form)

So how does it work?

It's time to move from theory to action, from ideal suppliers to real ones. I think everyone who is interested in already guessed that the XML that we got, we will convert in a cunning way, to get 50 from one format. For this we will use a two-step conversion:
Format converter
First, we need to create the data structure itself. XML is best for this because we have a powerful XSLT (eXtensible Stylesheet Language Transformations) language that allows you to convert an XML document from one type to another.
The language is not complicated, it can be mastered at a basic level in a couple of days, and 99% of my XSLTs cost only two instructions value-of and for-each.
Further, the created structure may need to be converted to a format other than XML. In this sense, the most difficult is the transformation of XML <-> JSON, an example of this transformation, I spread. (but you will not show, because the UFO has forbidden to insert here the link to InfoStart) There is still an XML conversion to the parameters of the GET request, i.e. in the line of the form? item = 12345 & qty = 4 & price = 19.5? but this is quite simple. As a result of this step, an exchange text, valid for a specific vendor, should appear.
After receiving a valid text for exchange - it remains only to send. This task is handled by the functionality, which I provisionally call the universal dialer. It can adjust the SOAP or REST service by its settings and transfer the result of the converter operation there. The dialer should provide very different scenarios for the development of events: it should be ready to send data as parameters both as request body, POST and GET methods, with basic and not very authorization, etc.
Theoretically, the Dialer can be screwed at least reading /writing in the Excel, specific "connectors" choose according to their needs.
In principle, the functions of obtaining data in the format of the ideal 51st supplier, XSL transformation, format converter and dialer can be taken out in a separate instance, without linking this functionality to any of the existing systems, which I have done. I call this Unit Interface Converter.
The interface converter code can fit into 300 lines. The basic settings (XSLT texts, conversion and call settings) are stored as data. As a result, the process of sending an order to a vendor is as follows:

ERP forms the order and sends it in the format of the 51st supplier to the Interface converter
The converter receives an XML test in the format of the 51st supplier and enriches it with
determines which vendor to send the request (conditionally, by the interface code), - performs XSL transformation and format conversion
on the fly.  
immediately sends the converted request to the supplier
receives the response of the supplier
from the body of the answer in the reverse order performs the format conversion, re-enrichment, then XSL transformation into the answer format of the ideal supplier
returns the response of the ideal supplier
to ERP.  
All this happens "on the fly", i.e. synchronously, i.e. ERP thinks that it is exchanging with all suppliers in one convenient format, without even noticing that something is going wrong. This approach, of course, is applicable only for cases of transmission of relatively small data packets. If you have price lists for 1 GB - it's worth considering how to tie asynchronous data to the whole story.

The chip with additional enrichment

Almost in every exchange there are values ​​that are simply hardcoded. In part, this can be a consequence through the rich functionality of the vendor, for example, the parameter "the percentage for which the price of the goods can be increased without additional agreement", in part it's just some values ​​like "for you it's always 4000". What is 4000? Why 4000? Just 4000 and that's it. In order not to store such "dead" values ​​in ERP, the Converter has the opportunity to enrich the ERP data with some other XML. Those. it's just arbitrary XML, which is stored in the interface converter DB.
For each interface (provider), such XML is its own. Enrichment occurs on the principle of simple gluing together with a wrapper into a common tag, for example:
& ltbody & gt;
& nbsp & nbsp & ltERPData & gt;
& nbsp & nbsp & nbsp & nbsp; data from ERP
& nbsp & nbsp & lt /ERPData & gt;
& nbsp & nbsp & ltAdditionalData & gt;
& nbsp & nbsp & nbsp & nbsp; static converter data
& nbsp & nbsp & lt /AdditionalData & gt;
& lt /body & gt;

And further, such enriched XML goes to the input of the XSLT engine, where both the data sent by the ERP and our static data are available.

What do we have as a result?

We're not just starting to start all this garden, but because we wanted to reduce the time for connecting the supplier. That's what the connection of a new supplier is:
Study documentation for the API
Create an appropriate vendor in ERP and configure it (about 10 checkmarks).
Description of the transformation of the exchange text in the language XSLT
Description of format conversion (if required)
Configuring outgoing connection settings (10 more checkboxes).
In practice, all these actions take up to 3 hours. It is clear that there are cases when the supplier will invent something "such", but in 95% of cases you can meet in 2-3 hours. (in this sentence I made a mistake and wrote a "support" instead of "supplier." Coincidence? )
Naturally, there are a lot of details and nuances that can not be described in the framework of the review article, but the goal of the article is to show one possible approach to designing the architecture of the multi-format exchange, I believe.
+ 0 -

Add comment