This is the English translation of Robuster Daten-Import/Export in Odoo auf Basis von connector_flow.
An ERP system is normally not a self-contained IT system, but instead incorporated into a multitude of processes that require interaction with other systems, for example the import of the product master data of an Excel catalogue, the transfer of orders to a logistics service provider via FTP or enabling a bank to import account movements. Although other programs can indeed access Odoo (for example via RPC or REST-API), a solution that can be directly implemented in Odoo is normally the easiest solution for all tasks that extend past pure data conversion.
This is where our Odoo module connector_flow comes into play.
It provides a structure in Odoo in which recurring tasks can be implemented and which handles standard tasks such as FTP download/upload and CSV processing. Individual procedures can be implemented separately and combined to form one process via defined interfaces. connector_flow is based on the excellent tried and tested Connector Framework, which facilitates a checkable, traceable and parallelisable workflow.
The following screenshots provide an example of a greatly simplified import of product master data from a CSV file. We also used the same technical basis to create an import that reads product data for approximately half a million products from a CSV file for two of our clients.
Our product import can be divided up into two stages: we first import the CSV file and then manage the products in Odoo. The associated technical process (Task Flow) is therefore comprised of these two tasks.
When carrying out the first stage, namely handling the CSV file, we are able to use a task that is already provided with the connector_flow module. This csv_import splits a CSV file into its separate lines and stores each of these individual lines as a Python data structure (a list or dictionary), also known as a chunk.
In the second stage, the creation of the product in Odoo, we can then add or update a product from such a chunk (a line of the CSV). For the technical implementation, we need to both program this task and compile the Task Flow.
We first deal with the task required for the creation of the product. The fact that this task receives a chunk from the previous task (reading the CSV file) as an entry means that we inherit the class abstract_chunk_read_task from the connector_flow module. In the chunk-data parameter we receive a dictionary so that we can suitably configure the reading of the CSV. The keys of the dictionary correspond to the column names of the CSV file (example files), meaning that creating the product in Odoo on the basis of the chunk is just a matter of a few clicks:
class product_catalog_import(abstract_chunk_read_task): def read_chunk(self, config=None, chunk_data=None, async=True): product_data = { 'name': chunk_data.get('Name'), 'list_price': float(chunk_data.get('Preis VK')), 'standard_price': float(chunk_data.get('Preis EK')), } product_image_url = chunk_data.get('Image URL') if product_image_url: url_obj = urllib2.urlopen(product_image_url) product_data['image'] = b64encode(url_obj.read()) self.session.create('product.product', product_data) |
In order to use our product_catalog_import class in the Task Flow, we still need to register it in the Odoo model impexp.task. This is carried out using the normal Odoo inheritance process. On the one hand, an identifier needs to be added to _get_available_tasks and on the other hand, a method …_class needs to be created and given a name that corresponds to the selected identifier.
class product_catalog_import_task(orm.Model): _inherit = 'impexp.task' def _get_available_tasks(self, cr, uid, context=None): return super(product_catalog_import_task, self) \ ._get_available_tasks(cr, uid, context=context) \ + [('product_catalog_import', 'Produkt Catalog Import')] _columns = { 'task': fields.selection(_get_available_tasks, string='Task', required=True), } def product_catalog_import_class(self): return product_catalog_import |
This enables us to set up our Task Flow. Although this can also be carried out manually via the Odoo interface, a configuration via XML is recommended in the interest of clarity. When completing this configuration, we create records for the Task Flow, the two tasks and a transition between the two tasks. The first task in the Task Flow needs to have the flow_start property. In our case, this is the csv_import, which comes with the connector_flow module. We then configure it using the includes_header property in the config value so that it uses the first line of the CSV file as a title, which it in turn uses to create dictionaries from the remaining lines. The second task is the product_catalog_import in accordance with the indicator that we chose above in the impexp.task model. We then finish off by adding a transition between the tasks.
<!-- Task flow --> Product Catalog Import <!-- Task and task transition --> Product catalog CSV to chunks csv_import <![CDATA[{'includes_header': True}]]> Product catalog chunks to products product_catalog_import |
To carry out the product import, we use the wizard available under Connector > Import/Export > Run Task. We select „Product Catalog Import“ as the Task Flow and our CSV file as the file.
Carrying out the product import
Example files for the product import
Shortly (how shortly depends on the configuration of the connector framework) after Run Task has been clicked, three products have been added to Odoo; the fourth product, „orange“, from the input file is missing.
The example products, apple, kiwi and lime, have been imported.
Under Connector > Queue > Procedures we can see that the execution of the second task has failed because the image URL is incorrect. In order to still be able to import the product, we can click on the Related button to edit the raw data and, for example, correct or delete the URL. After this data correction, the procedure can be started again so that the orange is also available in Odoo.
A line in the CSV file contains incorrect data.
The source code of the connector_flow module is available on Github, where you will also find the example code presented above under the name of connector_flow_example_product together with an example CSV (contrib/fruit_catalog.csv).
If you are also interested in Odoo/Open ERP or require our expertise for your ERP project, please feel free to contact us. We will be happy to help you further! You can contact us by calling +49 4105 5615699 or by sending us an e-mail at [hide_email sales@initos.com].