How we automated and sped up data collection by a magnitude

If you run a business then you have a process. It’s the set of activities you have to do to get your product or service to your client. Good Businesses have good processes, great businesses use business process automation. Business Process Automation is the use of technology to automate manual, repetitive tasks so that employees can focus on more important business tasks.

While that all sounds nice, why do businesses do it? What’s there to gain by spending money on improving how employees do their jobs? Why are the aforementioned Great businesses so focussed on it?

Well to quote one of our Clients: “The custom web-based system (…) allows efficient centralized management of our entire product universe. No license fees are a very nice bonus.”

There are many advantages to improving your operations including

• Reduced operating costs, leading to more profit
• Increase in quality of your goods and services, which leads to more profit
• Increased employee productivity, which leads to happy employees and, you guessed it, more profit

So assuming you’re on board, how would you proceed?

Well, when automating a business process there are two roads you can take. Either you keep the process as is, swapping certain manual steps for a software solution, or you make the brave decision to first analyse the needs in the context of the bigger picture and then create something great.

The latter involves taking a deep dive into the workflow to see what are the actual problems, and what is just a coping mechanism due to the design drawbacks of the current way. The goal is to throw away what doesn't work and introduce new fitting ideas, in other words it’s like reaching the same destination by taking a different path. As you can imagine, the redesign is an entirely different beast as compared to the straightforward automation, but most often yields the better results.

Today we want to look at a solution we recently delivered to a client, how it affected their day to day operations and the impact of taking the road less travelled.

Let’s first introduce our client.

Our Client - Score Media Group

Score Media Group is Germany's leading marketer for over 420 regional daily newspapers, their associated news portals and free weekly periodicals. They coordinate adverts and print runs with their huge network of big german newspapers like Hamburger Abendblatt, Der Tagesspiegel or Kölner Stadtanzeiger, to reach millions of people every day.

Score Media Group Business Needs

The same property of a newspaper may change over time, for example it might get printed on a bigger scale due to increased popularity, and only the most recent value of that property is of interest.

To stay up-to-date on the properties of the marketed products Score Media Group performs regular customer surveys. This consequently results in large amounts of information which needs to be collected, stored and processed daily. Score Media Group reached out to us, with the goal to automate and improve this knowledge management process.

Product Information Management System

What Score Media Group looked for was a Product Information Management (PIM) system, tailored to their business specifics. PIM is basically a centralized hub, storing product information. As such it could be an extremely valuable tool across many departments, including research and development, manufacturing, marketing, and sales.

A correctly configured normalized system would ensure that everybody has the same knowledge of the products, and thus can make clear and justifiable choices. Going a bit further would bring us to business intelligence - a process for making data-driven educated decisions, which is impossible without having proper data.

At the kick-off meetings, we identified the two essential requirements to make information gathering successful: Surveys need to reach the right people in the huge partner network and be easy to fill out.

Is Salesforce not enough? Salesforce is Customer Relationship Management (CRM) software, which can be extended to store product information as well. It can be internally programmatically adapted to your business needs with the use of Flows, Lightning Components and APEX classes, or alternatively it can connect to external apps via an API.

The Solution

As Salesforce has already been in use by some of the departments, extending it to PIM was the most logical way forward.

With the data storage question answered, the next question was how do we handle data gathering. Should it be done directly in Salesforce or in an external system?

An obvious choice is to do it directly in Salesforce by giving customers partial access via the Salesforce Community Cloud. After evaluating the consequences of this choice, we decided against this due to the scaling of licence costs with each additional user account and the limited possibility of user interface design. Instead we went for a custom Django web application, connected to Salesforce. While the initial costs of development and hosting were higher, they were greatly offset as the number of users increased without ballooning licence fees. Additionally, the gained flexibility allowed us to solidly implement a smooth login process, automatic email reminders and reporting functionalities.

Something we did not fully appreciate in the beginning, but have been quite happy about in retrospect, is that having two separate systems allows for a nice separation. While releases on Salesforce have to be coordinated with all involved parties, the web application can be released quickly and independently. This allows for quick iterations and fast development.

However, since there is no such thing as free lunch, everything comes at a price. Reliable syncing between two systems is not trivial and increases complexity. Behind the scenes, contact and product information needs to be transferred from Salesforce to the web application, and gathered product information has to be synchronised back to Salesforce.

While we did our best to make it look like it just works magically, there was quite a bit of complexity involved in implementing this bi-directional sync. For the developers or generally curious readers, check out the nitty gritty details in the next section. However, if you couldn't care less or want to keep the magic alive, feel free to skip.

The Nitty-Gritty

So how does the solution work in the end? Employees of Score Media Group can easily open a new customer survey in Salesforce via a Salesforce Flow, specifying the information they want to inquire about the selected products.

The Apex code behind the scenes selects the recipients relevant for the survey. For example a survey about marketing possibilities needs to reach a different person than one about financial reports.

Once the survey is confirmed in the final submission screen, a request is sent out to the REST-API endpoint of the Django web application. This transfers relevant information like contact emails, newspapers and questions to the web application.

The web application creates the survey and returns its id and URL back to Salesforce. Both are saved in Salesforce such that Score Media Group gains a good overview of current, planned and past surveys.

After the survey has been opened, the web application sends out invitations to the recipients. Once authenticated, they can fill out the survey in a tabular UI powered by React. The information can be updated while the survey is open. Once the survey is closed, it can not be changed anymore. Users can view and download a report with the answers they gave at any time.

How does the information now get back to Salesforce? There are a couple of ways to integrate web services. We chose to let Salesforce handle it via API calls to the Django web application.

The API endpoint returns the data in chunks to keep the response payload small even when there were a lot of changes since the last sync. Hereby we process all the chunks until the sync is complete and no data is returned from the API anymore. We schedule the Queueable Apex process to run hourly in order to frequently sync answers between the Django web application and Salesforce.

The endpoint is clever in the sense that when the same information is changed several times in between syncs, it gets reduced to the latest value, preventing unnecessary data processing.

The fetched information is saved into corresponding Salesforce objects. We designed the database structure to provide clear relationships and data ownership. All sales and CRM related information is owned by Salesforce and even uses the standard objects such as Account, Contact and Product2, which allows for an easy integration into the rest of the Salesforce world. The gathered additional information is clearly separated and implemented as custom objects representing the survey, questions and answers. One single object, ProductTrait, provides the link between Product and survey answers. This allows for a nice enhancement of the product list or detail page with custom information.

The Impact

Coming back to the original questions, Was it worth it? What did Score Media Group gain by taking the road less travelled? Did they accept the solution? Was the data collection really sped up? Was it even a good idea?

Well we could toot our own horn but we feel it would be better to hear it from them:

Tamara Bruylandt, Head of Product & Media Score Media Group:

"As a marketer of more than 420 regional daily newspapers, the associated news portals and free weekly newspapers, we were looking for a solution to quickly and easily retrieve product information from our publishers. The custom web-based system developed together with the Djangsters provides direct access to all levels of our publishers and allows efficient centralized management of our entire product universe. No license fees are a very nice bonus"

Linda Neumann, Senior Client Service Manager, Score Media Group:

„Whether questions about the portfolio or a wide variety of formats. It used to take us a very long time to retrieve product information from our many member companies individually and compile it manually in an Excel spreadsheet. Not only did this take a long time, it was also prone to errors. Now we can use the web-based dashboard to record and manage data quickly and export it with just a single click. Compared to before, we have made a quantum leap with PIM - which stands for Product Information Management”

In short we improved operations by orders of magnitude without inflating operating expenses with extra license fees. We reduced the amount of work employees had to do while increasing data integrity and giving people across departments access to up-to-date information.

The Conclusion

In the end, the solution was a resounding success. This was possible because we invested time in finding a fitting solution for our client Score Media Group. We planned each charted course, took careful steps along the byways, but more, much more importantly, we did it our way.

Not only did we leave our client in a better place, we gave them a solid foundation to expand on. What comes next? The sky's the limit, but a host of processes can be automated by utilising the collected information for more efficient downstream analysis and reporting. We are excited to support our client on the journey down this road.