Dimitri Gatowski
e-commerce architecture, web technology
E-commerce architecture
Magento 2 Preview
A couple of days ago I wrote my first blog post about my experience at the Magento Developers Paradise 2011 on Ibiza and promised to dig deeper into the details of Magento 2, which I noted while listening to the talks by Yoav Kutner, Dmitry Soroka and Anton Makarenko. I think that this project is crucial for Magento to keep and develop its most precious asset: the Magento community. The Magento 2 project will show what the acquisition of Magento by Ebay really means for us. The release date is planned for mid/end of 2012.
The first thing I want to mention is that Magento wants the development process be more transparent to the community. Therefore Magento will soon provide view access to the development tool stack, which consists of following Atlassian products:
Jira – issue and project tracking
Confluence – wiki for documentation
Bamboo – continuous integration
Crucible – code review tool
Fisheye – repository browser (thanks for the screenshot by Andreas von Studnitz in his blogpost about Magento 2)
Jira should give us a much better overview of the currently developed release and the planning of further releases. Confluence will provide a wiki with the Magento documentation. Bamboo should show metrics like broken builds ratio, unit tests count and code coverage percentage. Making Crucible public will provide a completely transparent code review process. I can't really believe that this will be made public, because it is very intimate ;). And Fisheye will make browsing the code and looking on diffs between version smoother than now (actually you can only browse the directory tree in SVN and just look into files).
It was also interesting to hear that the Magento core team consist of 6 developers and is using Scrum.
Magento was and is often critisized that Magento lacks of automated tests and proper documentation. And in deed this leads to unstable releases and code which is hard to understand. At the moment maintaining Magento solutions is more expensive as it should be. The time to qualify a PHP developer to be able to work with Magento is quite long (approx. 6 months which is not only my opinion) and expensive. This makes it harder for Magento solution partners to scale their development capacity. In addition, the lack of automated tests makes it harder to develop and maintain customizations and even the core itself. Clients expect a stable solution and low maintenance costs from their Magento solution partner. Magento 2 is supposed to address these issues:
there are already approximately 1900 automated unit, integrity and performance tests for Magento 2
they already cover 40-50% of the code
the documentation process (inline and also external in Confluence) is implemented in the development process of Magento 2
there won't be features which are released without automated test coverage and documentation
Magento tries to cut the time to qualify a developer by 50%
Magento considers porting the tests down to Magento 1 (I'm doubtful about this one)
This is nice to hear, but I recommend Magento to fulfil these announcement for Magento 2. I remember when I have visited the Magento office in Kiev for the first time in the mid of June 2009. I've discussed lots of topics with the team around Dima, Micha and Anton and we also talked about unit tests.
I'm the guy on the left, Anton, Micha and Dima from Varien from the right (2009)
A couple of weeks after the trip I saw this tweet: http://twitter.com/magento/status/2506514713 and was very excited about this news, but still we don't have automated tests today.
The external documentation in Confluence is also a very good point. There is no official publication about the Magento architecture in form of a book. The first literature I've discovered was a book written by Mark Kimsal in 2008: php/architect's Guide to E-Commerce Programming with Magento. It covers some basic architecture and digs into certain important modules. This was not really it, but it was better than nothing back then in 2008. In 2010 O'Reilly has published a German book called "Magento – Das Handbuch für Entwickler", which I have co-authored. It went deeper into the core architecture and provided much more examples of how to apply the knowledge to practical tasks and issues. Unfortunately analyzing the book reviews today show, that people still have issues transferring the theory into real Magento development. Until now I haven't heard of any other books which go deep into the core of the code. Most of the books concentrate on the installation, configuration and guiding admin users through the Magento Admin Panel functionality for the shop setup and the daily business of a Magento store after the Go Live.
Long story short: I'm very excited to see the official Magento documentation and hope that developers can benefit from it and that it can be used to educate Magento developers.
I've enjoyed hearing that Magento 2 will throw out Prototype and extJs and will use jQuery only. My team at next commerce recently completed a project which target was to create a frontend theme which is using jQuery only and doesn't contain any inline Javascript and CSS in the templates. It will be a component of our D2C software core and will be used in future D2C projects.
Magento 2 will have unlimited theme fallback: this means that you can build an unlimited theme inheritance tree which allows avoiding code redundancy in the templates, CSS and JS files completely. In Magento 1 there are only three levels of inheritance.
Example for multiple theme inheritance in Magento 2
A GUI design editor will provide functionality like moving blocks in layouts via drap&drop and it also will be able to export pages as one template file which will consist of all block templates and all states.The designer can edit that template instead of having to perform changes in lots of templates scattered in lots of different modules.
Since Magento 2 is first of all a refactoring release, there will be a lot of changes to the architecture to improve scalability, extendability and quality of Magento.
In Magento 2 there won't be any magic getters and setters. This means that the interfaces to all objects in Magento have to be strictly declared in the code. Also the data model definition will be declared, so that the database is no more the source of the data model definition of objects (this will actually have an impact on EAV). Together with the stricter documentation policy we should have finally a proper API documentation for Magento 2.
Currently, Magento objects like products, orders and customers are contaminated with things like controlling the persistance of their data (load, save, delete). There will be a strict separation of domain models (define which characteristics objects like products have) and service models (perform operations on objects).
If you install two Magento extensions which are rewriting the same model, block, helper, etc. from the core, one of the extensions won't work, because the first extensions wins when it comes to rewriting. In Magento 1 the only solution for that is to modify one of the modules to build an inheritance chain, so that the modules both work again (of course this only works if the extensions change different characteristics of a class). Magento 2 will allow to configure inheritance chains of models from additional extensions without code changes, which makes it easier to upgrade these extensions. This is a feature which OXID eSales already has.
To be able to disable functionality easier, Magento 2 will have less coupling between modules. Magento 2 introduces components, which consist of tightly coupled modules: Sales (Sales, Checkout), Catalog (Catalog, CatalogIndex, CatalogSearch, …), CMS. These components can be disabled without breaking the rest of the system, so it will be possible to disable Sales to have just a browseable catalog and static content (CMS) or just only static content on the site, if you disable Sales and Catalog.
Modules in Magento 2 will contain all files which belong to the module in one folder: app/code/<pool>/<space>/<model>. Not only PHP classes, but also the templates and skin files will be in this path. The filesystem structure will be rearranged: there will be a folder called "pub", which will contain all files which should be accessible to the public (jslib, media, skins). The other directories (app, dev, lib, var) should not be delivered by the webserver directly. See a screenshot in Alexander von Studnitz post about Magento 2.
Objects (models, helpers, blocks, etc.) will be instantiated using this pattern: createObject('<class name>') (e.g. createObject('Mage_Catalog_Model_Product')) and not with Mage::getModel('catalog/product'). The method is of course still a factory which looks into the configuration for rewrites. Also the getTable-method will be changed: you will have to use getTable('catalog_product_entity') instead of using the alias getTable('catalog/product'). The method will return the table name which might have a table prefix.
These changes have an impact on the compatibility of Magento 2 to Magento 1: Magento 1 extensions won't work with Magento 2. It will be required to migrate extensions to be compatible with Magento 2. Magento will provide a documented migration path for that. I hope there will be a database migration tool, too.
Magento 2 will provide support for the RDBMS MySQL, Oracle and MS-SQL.The refactoring of the resource model is already announced for the upcoming CE1.6 and EE1.11 release and will make following changes to the architecture of the persistence layer of Magento:
Refactoring of the resource model layer in Magento 1
The advantage of the refactoring is that the non-RDBMS-specific code is in the Resource Model and everything which is RDBMS-specific is removed and put into a helper for each supported RDBMS. This makes it easier to introduce support for additional RDBMS with nearly no code redundancy. The helpers are only needed when it is required to go beyond the limits of the database abstraction layer provided by Zend_Db for better performance or usage of additional functionality which other RDBMS do not support. Also the install and upgrade scripts will be abstract (upgrade-0.0.1-0.0.2 instead of mysql4-upgrade-0.0.1-0.0.2) and only when it is required, Magento 2 will still support mysql-*, oracle-*, etc. scripts for RDBMS-specific code. This refactoring will be backwards compatible.
A breaking news for me was that Magento is considering to remove EAV and only provide the EAV database schema for the catalog as an optional mode. This would improve the performance significantly, because there will be less JOINs in the queries and therefore less temporary tables which have to be created to sort and filter such results in MySQL. Also Magento would require less indexing which makes data imports and modifications of data faster.
EAV was build in since the start to support the modification of data models for products, categories, orders, quotes, customers and customer addresses without changes to the database schema. So basically it was possible to change the data model of these entities during the runtime of Magento in the Admin Panel. I was involved in a couple of medium- to big-size ecommerce projects on Magento in my career and I can say, that such kind of functionality is not needed and it also can be very dangerous for the stability of the Magento solution. Changes to the data model have to be part of a release and have to go through a proper release process which consists of QA activities such as testing the changes on a staging environment. This requires to perform such kind of changes by using upgrade scripts to ensure that the data model of all entities is the same on all systems (dev, acceptance, staging, production). And since the changes are part of a release and have to be deployed it is not a big deal that there are changes to the database schema.
Everybody who has watched the evolution of Magento 1 has also noticed, that EAV was removed step by step from Magento to gain better performance: first the quotes were stored in flat tables instead of the EAV schema, then the orders followed. Order items were flat right from the start (if I remember correctly) to be able to generate sales reports without too much pain. The flat catalog was introduced to remove the overhead which slowed down the rendering of category and product pages. The flat catalog is a special kind of an index, where each store has its own category and product table.
For me the logical next step is to remove the EAV for customer and customer addresses and last but not least from the catalog.
Although Magento has often announced to extend the API to cover more functionality, the API is still way behind the possibilities you have in the Admin Panel. Magento 2 will address this issue by including the extension of the API into the development process of new functionality. So all new features will be released together with new API calls covering the new functionality. Also the API will support JSON-RPC and REST in addition to SOAP and XML-RPC which are supported in Magento 1.
There are a couple of things which should improve the maintenance of running Magento sites.
The upgrade process will have a better workflow: the upgrade will be performed on a staging copy of the store (code and database). This staging environment can be used for tests before deploying the upgrade on the production environment. Magento solution partners should all do this already now with own deployment scripts or manually on staging environments. If not: shame on them!
The Magento profiler will output profiling entries in a hierarchy and not as a plain list like in Magento 1. This makes the profiler output easier to read. Also the output format can be configured to support Firebug and other browser plugins.
At first I was expecting Magento 2 to go with the latest trends in the world of web applications. So called post-modern web applications (e.g. GMail) leave the classical request-response principle for loading complete HTML pages. They are designed like a native client-server application which has a powerful view running on the client and just fetches raw data from the server. I watched a nice presentation on the next09 by Malte Ubl in 2009, who was a Technical Director at SinnerSchrader at that time and now has joined Google in the end of 2010, with the title "Building post-modern Web Applications". It covers this topic and you can watch it here: http://video.nextconf.eu/video/923175/building-post-modern-web.
Today, I think that the approach of Magento to make a major refactoring release is more pragmatic than changing the architecture of Magento completely. It is possible to release earlier and remove a lot of urgent issues outlined above faster to help the community to deliver better solutions for clients, better extensions for the community and better feedback for Magento. The benefits of this will form a better foundation for a big step to a modern web application design. The big question is how the X. Commerce strategy will influence the evolution and especially the revolution path of Magento. Will Magento be able to perform such big steps being a part of X. Commerce?
Filed under Solutions Tagged with Bamboo, Confluence, Continious Integration, Crucible, EAV, Fisheye, Jira, Magento 2, multiple RDBMS, Refactoring, Scrum, Solutions, Unit Tests, X.Commerce
Magento Developers Paradise 2011 Wrap Up
Last week on the 4th of June 2011 I've started my trip from Hannover to Ibiza to attend the annual Magento Developers Paradise(MDP). I was already looking forward to the event because I have enjoyed my time at the Magento Developer Paradise on Mallorca in October last year. The atmosphere at this kind of locations is very nice and relaxed, everybody is in a good mood and enjoys the sun and free drinks during the breaks and after the talks.
Unfortunately my plane was late, so that I've missed my connection to Ibiza, had to spend some hours on the airport in Palma de Mallorca and missed the conference kickoff dinner. But on a MDP this is not a problem: even if you join at 11pm you still meet most of the attendees at the pool bar. So did I and had a chat with Dmitriy Soroka, lead system architect at Magento Inc., Thomas Fleck, CEO of Netresearch and host of the event, and lots of other people. Until 2am I've been chatting about Magento topics and what I expect of the upcoming Magento 2.
The next day started with having a breakfast together with Yoav Kutner, CTO of Magento Inc., and some other Magento core developers. After this at 9am Yoav held his keynote about the current status of Magento and what topics will be covered in the upcoming talks. The most interesting was of course Magento 2 which will be released mid/end of 2012. The development has already started, but not all decisions are final. The release will be a major refactoring release which will focus on software quality and performance. To achieve this, a lot of changes to the architecture will be performed, so that Magento 2 won't be backwards compatible with Magento 1 extensions.
Yoav Kutner is helding the keynote for MDP11
Right after the keynote Andrey Tserkus from Magento Inc. held his presentation "Breaking news: Magento starts running on multiple RDBMS". This feature will already be available in the releases Magento EE1.11 and CE1.6 next month. Magento will focus on the support of Oracle and MSSQL first, but will add more RDBMS in the future. This feature contains the refactoring of the resource model of the Magento core: in current releases there are only "Mysql4" resource models which contain MySQL specific code but also abstract database access by using Zend_Db functionality which allows using multiple RDBMS. In EE1.11/CE1.6 the resource models will be cleaned up and will only contain non-RDBMS-specific code. The RDBMS-specific code will be moved into resource model helpers. This enables to add support for other RDBMS with less effort since most of the code is in the abstract resource model. Only when it is required to leave the DB abstraction layer e.g. for achieving better performance you need to write code in the resource model helper for the specific RDBMS. The refactoring of the resource model is backwards compatible, so that custom Magento modules still will work using MySQL.
I have also enjoyed the following talks about checkout customization by Vitaliy Korotun (Magento Inc.) and extending Magento hosting capabilities by Philippe Humeau (nbs System). In the afternoon everybody was invited to join a boat trip to Formentera, a small and beautiful island next to Ibiza. We were on board of a catamaran so it was possible to go swimming and we had a delicious dinner on it, too. In the evening we were brought back to our hotel and you could meet plenty of attendees at the pool bar afterwards. Some of them went out to hit the bars and the clubs at night. The nightlife on Ibiza is amazing!
Catamaran trip
The next day had very exciting talks about Magento 2 by Dmitriy Soroka, lead system architect at Magento Inc., and "MVC – model lessons learned" by Anton Makarenko, member of the Magento core team at Magento Inc.. I will only sum up the, in my opinion, most important news on these topics. I will dig deeper into them and write down my thoughts in a follow up post about Magento 2.
transparent development process using the atlassian software stack and making that available for the community (continious integration, documentation, issue tracking)
automated unit, integrity and performance tests are available for Magento 2; there are already 1900 tests
the documentation will be included in the development process, so it is always complete and up to date
jQuery will be used in the frontend, no Prototype and no extJs
unlimited theme fallback, you can create unlimited theme inheritance trees
Magento considers the removal of EAV and only offer it as an optional mode for the catalog
domain structural integrity: there will be a strict separation between domain models and service models
removal of magic getters and setters enforcing strict data model definition and interface declaration for models
configurable inheritance chains to be able to resolve incompatibilities between modules; OXID eSales has this feature already
the Core API will be extended in parallel to the development of new features, so it will cover all functions; it will support SOAP, XML-RPC, JSON-RPC and REST
less coupling between modules: Magento will introduce components, which consist of modules which are tightly coupled; these components (e.g. Sales, Catalog) will be loosly coupled to the rest of Magento and can be disabled
I didn't mention lots of things and also was not able to write everything down during the talks, but I think this gives a good overview of all improvements which are planned for Magento 2. If Magento will be successful in realizing them, this will be a huge jump forward in software quality and transparency and this should distinguish Magento from other open source shop platforms.
At 8pm everybody gathered in the lobby for the night event. We went by bus to the historic city of Ibiza and had a guided tour to the fortification and the historic city of Ibiza, called Dalt Vila. After the tour we went to the famous Zoo Bar and had some drinks before getting back to the hotel. Like on the previous day, a large group of people bought club tickets and had an amazing night in the Privilege club.
Great view from the fortification in the historic town of Ibiza
On the last day you could chat to people sitting at the pool bar and spending some time before taking a cab to the airport. You could hear most of them discussing the news about the acquisition of Magento by Ebay and that Magento will be at the core of the new X.Commerce platform. Some people were sceptical, some were optimistic. I'm excited of how this will influence the Magento products and the community.
In the late afternoon I finally took a cab together with the team of Netresearch (we had the same flight from Ibiza to Mallorca) and had a good trip back home.
To sum it up: it was an amazing event with lots of good talks, interesting people and a very nice and relaxed atmosphere for networking. I can recommend this event and already plan to attend next year! I also wanted to thank Alice Patzer, Andrea Panajotowa, Annemarie Tappert and Thomas Fleck for organizing this event and Rico Neitzel for shooting lots of nice photos. You can see them here: http://www.flickr.com/photos/34764379@N04/sets/72157626906928668/
Filed under Solutions Tagged with Continious Integration, Developers Paradise, jQuery, Magento 2, MSSQL, multiple RDBMS, Oracle, Solutions, Unit Tests, X.Commerce
Magento - Das Handbuch für Entwickler
RT @BBK_Bund: Der #Warntag2022 läuft! Damit Sie eine Feedbackmöglichkeit haben, wir Erfahrungen gebündelt sammeln & die technische Funktion… 1 month ago
RT @newstoreinc: Together with @Adyen, we will soon bring Tap to Pay on iPhone to our customers. Brands will be able to accept contactless… 9 months ago
RT @tvanschadewijk: I'm excited that this week we're announcing that @gethighstreet becomes part of NewStore! This has been an amazing 7-ye… 1 year ago
RT @Henning_Witzel: We are looking for a Senior Product Designer and a Technical Writer at @newstoreinc. With retailers from all around the… 1 year ago
RT @glassmanamanda: Exploring the Supply Chain of the Pfizer/BioNTech and Moderna COVID-19 vaccines blog.jonasneubert.com/2021/01/10/exp… 1 year ago
Follow @dimitri_g
E-commerce architecture (2)
Spryker: rethinking e-commerce architecture with Project A
Digital Commerce Technology Update – Sep. 2014
Rethink e-commerce architecture
Magento and Ebay: 9 months later