Assistance with Open Source adoption

Open Source News

How to practice CI/CD the SnapLogic way

SnapLogic - Tue, 08/07/2018 - 14:27

With the SnapLogic Enterprise Integration Cloud’s (EIC) superior handling of different types of integrations for organizations, forward-looking companies are leveraging DevOps methodologies for their own data and application integration workflows and initiatives. CI/CD – continuous integration and continuous delivery – is a practice where code is built, integrated, and delivered in a frequent manner. This[...] Read the full article here.

The post How to practice CI/CD the SnapLogic way appeared first on SnapLogic.

Categories: ETL

How to Develop a Data Processing Job Using Apache Beam – Streaming Pipelines

Talend - Tue, 08/07/2018 - 12:21

In our last blog, we talked about developing data processing jobs using Apache Beam. This time we are going to talk about one of the most demanded things in modern Big Data world nowadays – processing of Streaming data.

The principal difference between Batch and Streaming is the type of input data source. When your data set is limited (even if it’s huge in terms of size) and it is not being updated along the time of processing, then you would likely use a batching pipeline. Input source, in this case, can be anything from files, database tables, objects in object storages, etc. I want to underline one more time that, with batching, we assume that data is mutable during all the processing time and the number of input records is constant. Why should we pay attention to this? Because even with files we can have unlimited data stream when files are always added or changed. In this instance, we have to apply a streaming approach to work with data. So, if we know that our data is limited and immutable then we need to develop a batching pipeline.

Things get more complicated when our data set is unlimited (continuously arriving) or/and mutable. Some of the examples of such sources might be the following – message systems (like Apache Kafka), new files in a directory (web server logs) or some other system collecting real-time data (like IoT sensors). The common theme among all of these sources is that we always have to wait for new data. Of course, we can split our data into batches (by time or by data size) and process every split in a batching way, but it would be quite difficult to apply some functions across all consumed datasets and create the whole pipeline for this. Luckily, there are several streaming engines that allow us to cope with this type of data processing easily – Apache SparkApache FlinkApache ApexGoogle DataFlow. All of them are supported by Apache Beam and we can run the same pipeline on different engines without any code changes. Moreover, we can use the same pipeline in batching or in streaming mode with minimal changes – the one just needs to properly set input source and voilà – everything works out of the box! Just like magic! I would dream of this a while ago when I was rewriting my batch jobs into streaming ones.

So, enough theory – it’s time to take an example and write our first streaming code. We are going to read some data from Kafka (unbounded source), perform some simple data processing and write results back to Kafka as well.

Let’s suppose we have an unlimited stream of geo-coordinates (X and Y) of some objects on a map (for this example, let’s say the objects are cars) which arrives in real time and we want to select only those that are located inside a specified area. In other words, we have to consume text data from Kafka topic, parse it, filter by specified limits and write back into another Kafka topic. Let’s see how we can do this with a help of Apache Beam.

Every Kafka message contains text data in the following format:

  id – unique id of the object,
  x, y – coordinates on the map (integers).

We will need to take care of the format if it’s not valid and skip such records.

Creating a pipeline

Much like our previous blog, where we did batching processing, we create a pipeline in the same way:

Pipeline pipeline = Pipeline.create(options);

We can elaborate Options object to pass command line options into the pipeline. Please, see the whole example on Github for more details.

Then, we have to read data from Kafka input topic. As stated before, Apache Beam already provides a number of different IO connectors and KafkaIO is one of them. Therefore, we create new unbounded PTransform which consumes arriving messages from specified Kafka topic and propagates them further to the next step:

pipeline.apply( KafkaIO.<Long, String>read() .withBootstrapServers(options.getBootstrap()) .withTopic(options.getInputTopic()) .withKeyDeserializer(LongDeserializer.class) .withValueDeserializer(StringDeserializer.class))

By default, KafkaIO encapsulates all consumed messages into KafkaRecord object. Though, next transform just retrieves a payload (string values) by new created DoFn object:

.apply( ParDo.of( new DoFn<KafkaRecord<Long, String>, String>() { @ProcessElement public void processElement(ProcessContext processContext) { KafkaRecord<Long, String> record = processContext.element(); processContext.output(record.getKV().getValue()); } } ) )

After this step, it is time to filter the records (see the initial task stated above) but before we do that, we have to parse our string value according to the defined format. This allows it to be encapsulated into one functional object which then will be used by Beam internal transform Filter.

.apply( "FilterValidCoords", FilterObjectsByCoordinates( options.getCoordX(), options.getCoordY())) )

Then, we have to prepare filtered messages to write back to Kafka by creating a new pair of key/values using internal Beam KV class which can be used across different IO connectors, including KafkaIO as well.

.apply( "ExtractPayload", ParDo.of( new DoFn<String, KV<String, String>>() { @ProcessElement public void processElement(ProcessContext c) throws Exception { c.output(KV.of("filtered", c.element())); } } ) )

The final transformation is needed to write messages into Kafka, so we simply use KafkaIO.write() – sink implementation – for these purposes. As for reading, we have to configure this transform with some required options, like Kafka bootstrap servers, output topic name and serialisers for key/value.

.apply( "WriteToKafka", KafkaIO.<String, String>write() .withBootstrapServers(options.getBootstrap()) .withTopic(options.getOutputTopic()) .withKeySerializer(org.apache.kafka.common.serialization.StringSerializer.class) .withValueSerializer(org.apache.kafka.common.serialization.StringSerializer.class) );

In the end, we just run our pipeline as usual:;

This time it may seem a bit more complicated than it was in previous blog, but, as one can easily notice, we didn’t do any specific things to make our pipeline streaming-compatible. This is the whole responsibility of the Apache Beam data model implementation which makes it very easy to switch between batching and streaming processing for Beam users.

Building and running a pipeline

Let’s add the required dependencies to make it possible to use Beam KafkaIO:



Then, just build a jar and run it with DirectRunner to test how it works:

# mvn clean package
# mvn exec:java -Dexec.mainClass=org.apache.beam.tutorial.analytic.FilterObjects -Pdirect-runner -Dexec.args=”–runner=DirectRunner”

If it’s needed, we can add other arguments used in the pipeline with a help of “exec.args” option. Also, make sure that your Kafka servers are available and properly specified before running Beam pipeline. Lastly, the Maven command will launch a pipeline and run it forever until it will be finished manually (optionally, it is possible to specify maximum running time). So, it means that data will be processed continuously, in streaming mode.

As usual, all code of this example is published on this github repository.

Happy streaming!

The post How to Develop a Data Processing Job Using Apache Beam – Streaming Pipelines appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

20 Years with, and at OSCON

Open Source Initiative - Tue, 08/07/2018 - 09:40


OSCON, the annual open source conference organized by O'Reilly Media, is always a great event for the open source community to come together to acknowledge the advancements of the open source software movement and the communities that enable it. However 2018 was a special year as several open source projects and communities marked significant milestones and celebrated anniversaries, including the OSI (you may have heard, it's our 20th).

In recognition of the success of the open source software movement, and successes of so many organizations that have contributed to software freedom, the OSI organized a full-day of presentations, discussions, and activities. The track, "Open Source Anniversary: Our Shared Successes", not only celebrated the founding of the open source software movement and the OSI itself in 1998, but also the anniversaries of several other key initiatives that have enabled the free and open source software movement to thrive.

  • Deb Nicholson, Director of Community Operations at Software Freedom Conservancy, presented 30+ years history of the Free Software movement.
  • Jose Parrella, Principal Program Manager at Microsoft and Debian Developer, provided highlights of Debian's 25 years.
  • The 25th anniversary of Red Hat was covered by Michael Tiemann, former OSI Board President and VP Open Source Affairs at Red Hat.
  • The FreeBSD Foundation, celebrating 25 years, was represented by their Executive Director, Deb Goodkin.
  • Abby Mayes, Practice Lead, Working Open at Mozilla, shared Mozilla's 20 year history.

Highlighting the continued growth and maturity of the now twenty-year open source software movement, our track also included several other talks featuring emerging initiatives:

  • OSI Incubator Project ClearlyDefined, crowdsourcing FOSS data for project success, was introduced by Carol Smith, Senior Open Source Program Manager at Microsoft and OSI Board Director.
  • Software Heritage, browsing 20 years of FOSS, and then some, was presented by Stefano Zacchiroli, Associate Professor of Computer Science at Université Paris Diderot, CTO of Software Heritage, former Debian Project Leader, and former OSI Board Member.
  • OSI Board Director/Treasurer and Program Manager at Google, Josh Simmons, presented, "Teaching the next generation to FLOSS."

Finally, we were very fortunate to host two very special keynotes:

Later in the evening, to celebrate this special occasion, we hosted a party, "Cupcakes & Cocktails" where the real highlight of the day took place: a memorable panel discussion on the founding the OSI, and its early work in promoting open source, with OSI Co-founder Bruce Perens, former OSI President Michael Tiemann, and former OSI DIrector Danese Cooper, moderated by current OSI President Simon Phipps. We were honored with over 150 guests joining us for the party, who not only heard some of the history of open source from those who made it, but may have also participated in a little bit of history themselves that evening.

While the anniversary track was a real highlight in our celebration of open source over the past 20 years, throughout OSCON we also conducted interviews with open source luminaries to capture their recolections of the movement's beginnings, and expectations for the future of open source software. We were fortunate to speak with Tim O'Reilly, Bruce Perens, Chris DiBona, Luis Villa, Alolita Sharma, and several other key people from the open source movement. In the coming months we'll share these interviews, along with each of the presenttions and the panel discussion from the day's events, here and on our community portal OpenSource.Net.

Importantly, we want to thank our dedicated sponsors for their support in making all of our events, and indeed, all of OSCON so successful: Amazon Web Services, DigitalOcean, GitHub, Google, IBM, Indeed, Heptio, Linux Foundation, Microsoft, and Percona.

OSCON was a great opportunity to celebrate our Anniversary. We want to thank everyone who has helped us make this celebration a wonderful experience!



Categories: Open Source

Looking Beyond OAS 3 (Part 1)

Talend - Mon, 08/06/2018 - 10:39

After reviewing the history of the OAI and API specifications last year, I wanted to take a look at the future of the OpenAPI Initiative (OAI) and outline my calculated, plausible evolution paths for the project moving forward.

State of the OAI

Over the past year, we’ve hit several milestones. We have released the OpenAPI Specification (OAS) version 3.0, a major revision achieved by the long and intense work of the technical community with the support from the steering committee (TSC). AOI is now 30+ members strong and, we keep gaining new members. We are also ramping up our marketing efforts as highlighted by the transition of the API Strategy & Practice conference RedHat/3Scale to the OAI, under the Linux Foundation umbrella.

These are impressive achievements to attain in a short amount of time and we need to build on this momentum as the support of OAS 3.0 by various API tools is quickly expanding this year. Now, what are the next opportunities in front of us, where should we collectively focus our energy, so we can enable the Open API ecosystem to strive and expand?

Based on some recent discussions around the potential evolution of the OAS scope and our mission to help describe REST APIs, there are several compelling opportunities from which one should carefully choose. We’ll cover these options in a series of blog posts over the next few weeks, but this first post will focus on the opportunities around web API styles.

Web API styles a) Enhanced support for Resource-driven APIs

With the active involvement of the creators of RAML and API Blueprint within the OAI, version 3 of OAS has added many enhancements including better examples, modularity and reuse. These capabilities are essential -especially when collectively designing a large set of API contracts – and helped reduce the functional gap with alternative specifications. We could continue to look at ways to close this gap even more so now that OAS eventually becomes a full replacement for RAML and API Blueprint in the future.

I grouped these three API specifications in a Resource-driven category because the notion of a Resource, identified by a Uniform Resource Identifier (URI) and manipulated via a standard set of HTTP methods (Uniform Interface), is central to the REST architecture style that they get inspiration from.

It is worth mentioning additional specs in this category such as the JSON-API that propose conventions to define resource fetching (sorting, pagination, filtering, linking, errors, etc.) and can help improve consistency of REST APIs. With OAS 3 capabilities, it should be possible to capture those conventions in a base OAS file reusable by several concrete API contract files (see the $ref mechanism). It’s feasible to see the OAI supporting the JSON-API spec as an additional project next to the core OAS project one day in the future.

Projects like HAL, JSON-LD and ALPS can help experienced API designers implement long-lasting APIs by reducing the coupling between clients and servers by better supporting the hypermedia principle in REST, at the media type level. They are rather complementary with OAS and could also be candidate as additional OAI projects to increase its support for Resource-driven APIs.

Support for streaming APIs based on the Server-Sent Event (SSE) media type, compatible with REST, are also becoming more and more frequent and could be better described in OAS with potentially limited changes.

The way OAS describes web APIs couple clients and servers at the resource level, isolating the clients from the lower level server implementation details such as the more complicated underlying microservices or databases, covering a very broad range of use cases that made it widely adopted in our industry.

However, there are situations where a web API is essentially a Function-driven API or a Data-driven API as illustrated below. Let’s now explore what that means and how the OAI could help describe those APIs in a standardized way.

b) Support for Function-driven APIs

For function-driven APIs (aka RPC APIs), a set of functions (aka methods or procedures) with their custom name, input and output parameters constitute the central API piece that is described in the contract. They also tend to rely on HTTP as a simple transport protocol, ignoring its application-level capabilities to offer more direct binding to programming languages that are in the majority function-driven as well.

While W3C SOAP was the most widely deployed variant due to its popularity 10 years ago as part of the SOA (Services Oriented Architecture) and WS-* bandwagon, it was quite complicated technically and resulted in low performance and limited support outside the major two programming environments (Java and .Net).

Because of its shortcomings, many RPC alternatives were developed such as XML-RPC, JSON-RPC that were already simpler than SOAP. However, the latest trend replaces textual data serialization format by more efficient binary formats such as Apache Avro or Apache Thrift.

Today, the leading RPC project is gRPC, that was created by Google and donated to the Cloud Native Computing Foundation (CNCF), also part of Linux Foundation. It is successful in high-performance microservices projects due to its optimized data format based on Protocol Buffers also created by Google and its reliance on the high-performance HTTP/2.0 protocol. It would be an interesting development if the gRPC project became part of the OAI or at least if OAS was supporting the description of such APIs (aka gRPC service definitions).

If you are interested in this idea, there is already a related OAS request for enhancement.

c) Support for Data-driven APIs

Finally, there is a category of web APIs where a data model (or data schema) is the central piece that can be almost directly exposed. In this case, the API can be generated, offering rich data access capabilities including filtering, sorting, pagination and relationship retrieval.

Microsoft was the first to develop this idea more than ten years ago, with Google pursing and then retiring a similar project called GData. OData is based on either Atom/XML or JSON formats and is now a mature project supported in many business tools like Excel, Tableau or SAP. It also exposes the data schema as metadata about the OData service at runtime, facilitating the tool discovery. It can be compared to standards like JDBC and ODBC but more web-native.

Recently, alternatives to OData have emerged, first from Netflix with its Falcor project which lets JavaScript clients manipulate a data model expressed as a JSON graph. Facebook has also released GraphQL in 2016 and received a very good level of interest and adoption in the API developers’ community.

Interestingly, OData automatically exposes a resource-driven API and a function-based API at the same time, offering a hybrid API style. GraphQL also supports exposing custom functions in a function-based manner but doesn’t have resource-based capabilities, instead reimplementing its own mechanisms such as caching.

There is a need for this style of APIs in the ecosystem and OAI could support the description of API contracts based on data models compatible with GraphQL and OData for example (see this OAS issue for more discussion) and potentially even host such a project if there was a joint interest.

Wrapping up

Beyond the incremental updates that the community will naturally add to the OAS 3 specification (see the list of candidates here and please contribute!), I believe that the original scope of OAS could be extended to support additional web API styles. In the second part of this blogpost series, I will explore the common API lifecycle activities and how OAS could be extended to improve its support for them.

The post Looking Beyond OAS 3 (Part 1) appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

Extending Liferay DXP - User Registration (Part 1)

Liferay - Sun, 08/05/2018 - 20:19

I am starting a short series of blogs in which I will show how to customise User Registration / Customer Acquisition process in Liferay.

The inspiration for this blog is one of our DXP portal implementations which had very broad (and frequently changing) user data collection requirements.

There are many scenarios in which you want to customise user registration and account 
creation process:

  • creating a custom user registration form allows delivering a more consistent user experience;
  • when based on information a user provided an approver makes a decision to grant or reject the registration;
  • apart from default fields you can add any user account fields;
  • you can collect any other information related to your business and store it in your custom tables;
  • different registration flows might be required for different groups of users or sites;
  • a wizard-like registration form can deliver more information and help user to make the right choices;
  • users can apply for different roles or permissions at registration;
  • and it could be a place where you might want to display your marketing material.

For now I plan to write three articles:

1. Custom registration process for a portal with multiple sites. 
The main features:

  • Check if a user already has an account
  • Show and allow the user to select which site to join
  • Collect additional information from the user and send email notification to interested parties (approver for restricted site) on form submission
  • Automatically create the user account on form submission if the membership type of the selected site is open, otherwise create the user account but make it inactive (will be activated by approver)

2. Getting more data out of registration form 
The main features:

  •  Add fields to registration form that will be stored in User Account details
  • Collect information to populate user Custom fields
  • Add fields to store in custom tables
  • Allow users to download and print the entries they made to the form for their records

3. Taking payments at registration 
The main features:

  • If payment is required as part of request to join a site, display amount to pay and collect the payment
  • Allow users to download Receipt for their records


The Registration form will be based on SmartForms. Of course, you can built your own form/ui  but here are the reasons why you might consider Smartforms:

  • Flexibility - non-technical staff can edit the form and change wording to adjust or add explanations/instructions/conditions on the form.
  • Styling made easy - fully separated from software code and can be done by web designer.
  • Simple Integration with your other systems via webservices to bring data to the form.
  • Easy field value extraction from the submitted form into Liferay data stores or your other systems. 
  • But the most important - cutting costs on the porlet development, as form creation and further adjustments/changes can be done without software modifications.

You can download SmartForms from Liferay marketplace - or from SmartForms downloads page, additional information can be located here


Even if you are not going to use Smartforms I hope these blogs will help you as I will be providing samples of how to use Liferay services/APIs for your user registration purposes.

Victor Zorin 2018-08-06T01:19:00Z
Categories: CMS, ECM

Four ways an iPaaS accelerates Workday value

SnapLogic - Fri, 08/03/2018 - 16:40

Enterprises are flocking to cloud-based platforms like Workday to transform how they do business, yet integration and data migration challenges can trigger deployment headaches and unforeseen costs that impede time to value. Workday, which marries financial management with critical HCM functionality in areas including workforce planning, recruiting, talent management, human resource management, benefits, payroll and[...] Read the full article here.

The post Four ways an iPaaS accelerates Workday value appeared first on SnapLogic.

Categories: ETL

The new Web Experience functionalities released in 7.1

Liferay - Fri, 08/03/2018 - 05:52

In Liferay Portal 7.1 a lot of effort has been put in improving the experience of building sites and pages and, as a result, some new functionalities have been introduced to empower marketers and other business users in the creation of stunning websites.

We have achieved this by considering differentiated needs for the marketer, designer and web developer; and adding a new approach to content and page creation. We will continue working to expand the initiative in future releases, but some exciting features have already been introduced.

The key elements added to Web Experience under what we call Modern Site Building are Page Fragments, Content Pages, Display Pages and Multiple Navigation Menus. This is the first entry of a series of posts in which we will have the opportunity to go into more detail of each of those. But, as a starter, we will briefly introduce the big picture and how every component fits in it.


The context

Authors, marketers, designers, HR teams or whoever is in charge of creating a piece of content or site pages sometimes face repeat work, but not always, right? Sometimes they are happy to use web content structures because content reusability sounds like pure magic. Some other times they wish there was a faster way to create pages because it is a one-off and what really matters is minimizing the number of clicks while making sure it looks awesome. Put simply, the needs are not always the same.

Liferay has always used a very content centric and decoupled approach. The creation of content in the form of different assets and the display of it have always been two separate things. Thus, the long-established approach to page creation with Liferay is by combining content and applications. These pages are flexible, powerful and offer advanced functionality; but also add complexity to some use cases.


The authoring flow considered

For the new functionalities introduced we have considered a design and authoring flow like the one described below.


The marketer or the business user needs to create an effective landing page that communicates the desired message and increases customer engagement. The marketer does not have front-end knowledge but would like to have an interface to create site pages in a few clicks with editing options and making sure it looks great.

In order to achieve this, previous work of the Design and Web Development team is necessary. First of all, the designer designs several reusable page parts called Page Fragments (list of contents, headers, title sections, footers…).

The Web developer then implements these designs, coding the appearance and behavior of the Page Fragments and loads them into the Page Fragment collection library. Thereafter, the designer can create Page Templates by adding and combining Page Fragments, making a Page Template collection available for the marketer to directly start working on.

Using this approach, marketers will focus on writing content in Content Pages, using Page Templates created by designers which will consist of Page Fragments previously developed by web developers


Key new elements.

With this authoring flow considered, we have introduced new functionalities attending to the different roles and responsibilities in the creation process:

  • Page Fragments, which are collections of “design blocks” created by web developers using HTML, CSS and JavaScript that are made available to the non-technical users to build the Pages by composing them.

  • Content Pages, a new way of creating pages with a few clicks. The main benefits of content page are the easy edition of its content directly on the page.

  • Display Pages, which allow to control how a particular content is displayed in full page.

  • Multiple Navigation Menus, that leverages the Navigation by allowing to introduce several Menus on the header, on the footer, sidebar...


Themes with Page Fragments available from the Marketplace

To ease the Blank Page Syndrome you can start standing on some already defined examples. If you want to leverage on existing Page Fragment collections to start exploring the possibilities of the new features right away, you can already download the Fjord theme for Liferay Portal 7.1 from Liferay Marketplace. It contains a series of Page Fragments and Page Templates that can be used as a starter kit. More themes will follow soon!



I want to know more.

Read along the coming series of posts to learn about Page Fragments, Content Pages, Display Pages and Multiple Navigation Menus.

Also, if you want to learn more about how to empower your team to build modern sites, you can sign in for free to the lesson on “Building Engaging Websites” available in Liferay University or accessing Liferay Documentation.


Ianire Cobeaga 2018-08-03T10:52:00Z
Categories: CMS, ECM

New Project SDK Installers 3.2.0 GA1 Released

Liferay - Thu, 08/02/2018 - 21:45

We are pleased to announce the first general available release of Liferay Project SDK Installers that support Liferay 7.1.

New Installers:

For customers, they can download all of them on the customer studio download page.

The installer is the full fledged Liferay Developer Studio installer which installs Liferay workspace, blade, Developer Studio and comes pre-bundled with latest Liferay DXP server. It also supports to config a proxy using for download gradle dependencies.

Upgrade From previous 3.1.x:
  1. Download updatesite here

  2. Go to Help > Install New Software… > Add…

  3. Select Archive..., Browse to the downloaded updatesite

  4. Click OK to close Add repository dialog

  5. Select all features to upgrade, then click > Next, again click > Next and accept the license agreements

  6. Finish and restart to complete the upgrade

Release highlights:

Installers Improvements:

1. Better support HTTP, HTTPS, SOCKS5 proxy

 2. Bundle latest Liferay Portal

   - bundle 7.1.0 GA1 in LiferayProjectSDKwithDevStudioCommunityEdition installers

   - bundle DXP 7.1.10 GA1 in LiferayProjectSDKwithDevStudioDXP installers

Improvements for Deployment:

1. Support Liferay Watch Task

2. Support for Target Platform

3. Improve wizard to download dependencies in background

4. Better deployment support for Liferay DXP/7

   - support DXP 7.1.10 GA1 Tomcat and Wildfly

   - support Liferay 71 CE GA1 Tomcat and Wildfly

   - integration of Blade CLI 3.1.1

   - support Plugins sdk 1.0.19

   - support Liferay Workspace Gradle 1.10.2

5. Third party plugins update

   - update m2e to 1.8.3

   - updategradle plugin buildship to latest 3.0.0

6. Miscellaneous bug fixes


If you run into any issues or have any suggestions please come find us on our community forums or report them on JIRA (IDE project), we are always around to try to help you out. Good luck!

Yanan Yuan 2018-08-03T02:45:00Z
Categories: CMS, ECM

Making Data a Team Sport: Muscle your Data Quality by Challenging IT & Business to Work Together

Talend - Thu, 08/02/2018 - 11:29

Data Quality is often perceived as the solo task of a data engineer. As a matter of fact, nothing could be further from the truth. People close to the business are eager to work and resolve data related issues as they’re the first to be impacted by bad data. But they are often reluctant to update data as Data Quality apps are not really made for them or just because they are not allowed to use them. That’s one of the reasons bad data keeps increasing. According to Gartner, poor data quality cost rose by 50% in 2017, reaching $15 million per year for every company.  This cost will explode in the upcoming years if nothing is done.

But things are changing: Data Quality is now increasingly becoming a company-wide strategic priority involving professionals from different horizons. To succeed, working as a team like in a sport team is a fair analogy to illustrate the key ingredients to succeed and win any data quality challenge:

  • As in team sports, you will hardly succeed with a solo approach rather than tackling from all angles
  • As in team sports, there are some practice to make the team succeed and win
  • As in team sports, Business/IT Teams would need the right tools, the right approach, and the right people to tackle the data quality challenge

This said, it is not as difficult as one could imagine. You just need to take up the challenge and do things the right way from the get go.

  1. The right tools. How to fight complexity with simple but interconnected apps

There is a plethora of data quality tools on the market. Go and register to a big data tradeshow and you will discover plenty of data preparation, stewardship, and tools offering several benefits to fight bad data. But only a few of them cover Data Quality for all. On one side, you will have sophisticated tools requiring deep expertise for a successful deployment.

These tools are often complex and require in-depth training to be deployed. Their User Interface is not suitable for everyone so only IT people can really manage them. If you have short term data quality priorities, you will miss your deadline. That would be like trusting a rookie to pilot a jumbo jet with flight instruments that are obviously too sophisticated to end successfully.

On the other side, you will find simple and powerful apps that are often too siloed to be injected into a data quality process. Even if they successfully focus on the business people with simple UI, they will miss a big piece to the puzzle, collaborative data management. And that’s precisely the challenge: success relies not only in the tools and capabilities themselves, but in their ability to simply talk to one another. For that you would need to have a platform-based solution that share, operate and transfer data, actions, and models together. That’s precisely what Talend provides.

You will confront multiple use cases where it will be next to impossible to manage your data successfully alone. By working together, users will empower themselves through the full data lifecycle.  Giving your business the power to overcome traditional obstacles such as cleaning, reconciling, matching or resolving your data.

  1. The right approach

It all starts with the key simple steps approach to manage data better together: the right approach: analyze, improve and control.

Analyze your Data Environment:

 Start by getting the big picture and identify key data quality challenges. Analyzing will help to give the big picture of your data. Rather than profiling data on its own with Data profiling in Talend Studio, a data engineer could simply delegate that task to a business analyst who knows customers best. In that case, Data Preparation offers simple yet powerful features that help the team get a glimpse of Data Quality with inflight indicators such as quality in every Data Set Columns. Data Preparationallows you to easily create a preparation based on a data set.

Let’s take the example of a team wishing to prepare a marketing campaign together with sales but suffering from bad data in the SalesForce CRM System. With Data Preparation, you have the ability to automatically as well as interactively profile and browse business data coming from SalesForce. Connected to Salesforce thru DataPrep, you will get a clear picture of your data quality. Once you identified the problem, you can solve it on your own with simple but powerful operations. But you’ve only just scratched the surface. That’s where you would need the expertise of a Data Engineer to go deeper and improve your data quality flows.

Improve your Data with in depth tools and start remediation designing stewardship campaigns

 Using Talend Studio as your Data Quality Engine, the data engineers of your IT department will get access to a wide array of very powerful features included into Talend Studio. You will for example separate the wheat from the chaff using a simple data filter operation such as t-filter to identify wrong email patterns or exclude from your domain list improper domain addresses. At that stage, you will need to make sure you isolate bad data into your data quality process. Once filtering is done, you will then continue to improve your data and for that you will call on others for help. Talend Studio will work as the pivot of your data quality process. From Talend Studio, you will enable you to log on your credentials to Talend Cloud and expand your data quality to users close to the business. Whether you’re a business user or a data engineer, Stewardship-now in the Cloud will then allow you to launch cleaning campaigns and solve the bad data challenge with your extended team. This starts with designing your campaign.

Using the same UI look and feel as Talend Data Preparation, Talend Data Stewardship will offer the same easy to use capacities that business users love. As it’s fully operationalized and connected to Talend Studio, it will enable IT or Business Process People to expand Data Quality Operations to new people unfamiliar to technical tools but keen on cleaning data with simple apps relying on their business knowledge and experience.

That’s the essence of collaborative data management: one app for each dedicated operation but seamlessly connected on a single platform that manages your data from ingestion to consumption.

As an example, feel free to view this webinar to learn how to use cloud-based tools to make data better for all:


Control your data quality process to the last mile with the whole network of stewards

 Once you have designed your stewardship campaign, you need to call on Stewards for help and conduct the campaign to have them checked the data at their disposal. Talend Data Stewardship will play a massive role here. Unless other tools existing on the market, the ability to extend your data quality to stewards with UI-friendly applications will make it easier to resolve your data and make sure you have engaged key business contributors in an extended data resolution campaign. They will feel comfortable resolving business data using simple apps.

Engaging business people in your data quality process will bring your data quality processes several benefits too. You will get more accurate results as business analysts have the experience and required skills to choose the proper data. You will soon realize that they will feel committed and be eager to cooperate and work together with you as they’re finally the most concerned by Data Quality.

Machine learning will act here as a virtual companion of your data-driven strategy: as stewards will complete missing details, the machine learning capabilities of Talend Data Quality Solutions will learn from stewards and predict future matching records based on initial records resolved by Stewards. As the system will learn from users, it will give you free hands to pursue other stewardship campaigns and reinforce the impact and control of your data processes.

Finally, you will then build a data flow back to your SalesForce CRM System from your stewardship campaign so that bad data cleaned and resolved by Stewards will then be reinjected into the Salesforce CRM System. Such operations can only be achieved with simplicity if you have apps connected together on a single platform. You’ll have the opportunity to mark data sets as certified directly into a business app like Data Preparation so that users getting access to data will then have cleaned and trusted data to be used.

Remember this three-steps approach is a continuous improvement process that will only get better with time.s

To learn more about Data Quality, please download our Definitive Guide to Data Quality

The post Making Data a Team Sport: Muscle your Data Quality by Challenging IT & Business to Work Together appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

Oh no, my URLs disappeared…(and how to get them back)

Liferay - Thu, 08/02/2018 - 02:30

Recently we got a couple of complains about new Web Content article behaviour, specifically about the JournalArticle.getContent() method’s return value. The main problem developers experience is when they embed an Image into the Web Content article or use ddm-image or ddm-document-library field in their structures they expect to see the URL of the object(Image or D&M asset) in the raw XML when using JournalArticle.getContent() method, it actually was there in the 7.0 and the raw XML looked like this:

(...) <dynamic-element name="Image8r1v" type="image" index-type="text" instance-id="ryns"> <dynamic-content language-id="en_US" alt="" name="blonde.png" title="blonde.png" type="journal" fileEntryId="34506" id="34835"> /image/journal/article?img_id=34835&amp;t=1531817578959 </dynamic-content> </dynamic-element> (...)


There are two main differences in the 7.1:
We switched from the internal table JournalArticleImage to the common Documents and Media repository as a storage for the Web Content article images
DDM fields for Image and D&M assets changed their internal representation from the URL to the JSON object

Now the raw XML of the article with Images or ddm-image(ddm-documentlibrary) fields looks like this:

(...) <dynamic-element name="Image54q7" type="image" index-type="text" instance-id="wscg"> <dynamic-content language-id="en_US"> <![CDATA[\{ "groupId":"20124","name":"allatonce.png","alt":"", "title":"allatonce.png","type":"journal", "uuid":"80269faa-dea9-fd5a-cb78-3c7aa9da51ea", "fileEntryId":"36774","resourcePrimKey":"36772"} ]]> </dynamic-content> </dynamic-element> (...)


It was an internal decision and we didn’t realize that out there could be developers who actually use the raw XML content for their own needs…

First I would like to explain why it was done, not to try to excuse for this case, but to prevent such cases in the future. On the one hand, JournalArticle.getContent() method is a public API and its behaviour must be, at least, backward compatible, but on the other hand its behaviour depends on many components behind it, the signature of the method didn’t change and the implementation details (including the raw XML format of the content) never were published. To avoid such a problem we strongly recommend the developers to use published means for Web Content processing, such as JournalContent and JournalArticleDisplay. Both of them provide processed content of the Article without need to work with the raw XML, clear example can be found in the Web Content Display portlet:

(...) JournalArticleDisplay articleDisplay = _journalContent.getDisplay( article, ddmTemplateKey, viewMode, languageId, page, new PortletRequestModel(renderRequest, renderResponse), themeDisplay); String processedContent = articleDisplay.getContent(); (...) @Reference private JournalContent _journalContent; (...)


Also there is a taglib which allows to render a specific journal article using its JournalArticleDisplay instance:

<liferay-journal:journal-article-display articleDisplay="<%= articleDisplay %>" />


Or the developer can use JournalContent.getContent() method directly, the result must be the same - processed content where all the fields behave as expected.
Now let’s talk about how to get the URLs back because I understand that it could be a problem to refactor hundreds of lines of your code and the best way for the developers who use the raw XML would be to keep processing the URLs as they were doing it before.
Here I have to mention one detail - there is no way to return to the old format of the URLs for embedded images, so if you have some sort of regular expression catching “/image/journal/article...” - there is no way to make it work again.
There are two options to get the URLs back, both need to adapt your existing code, which works with the raw XML, a little bit.

First option is applicable when you have a concrete file entry ID:

(...) // here fieldValue is raw XML field value for your Image/DM field JSONObject jsonObject = JSONFactoryUtil.createJSONObject(fieldValue); long fileEntryId = jsonObject.getLong("fileEntryId"); FileEntry fileEntry = PortletFileRepositoryUtil.getPortletFileEntry(fileEntryId); String fileEntryURL = PortletFileRepositoryUtil.getDownloadPortletFileEntryURL( themeDisplay, fileEntry, StringPool.BLANK); (...)

And the second option is applicable in case when you don’t have specific file entry ID, but have UUID and group ID of the target entry:

(...) // here fieldValue is raw XML field value for your Image/DM field JSONObject jsonObject = JSONFactoryUtil.createJSONObject(fieldValue); String fileEntryGroupId = jsonObject.getLong("groupId"); String fileEntryUuid = jsonObject.getLong("uuid"); FileEntry fileEntry = PortletFileRepositoryUtil.getPortletFileEntry( fileEntryUuid, fileEntryGroupId); String fileEntryURL = PortletFileRepositoryUtil.getDownloadPortletFileEntryURL( themeDisplay, fileEntry, StringPool.BLANK); (...)


Hope these 5 lines of code help you to solve the problem. We understand that it could be frustrating to deal with such changes and we are trying to do our best to avoid them without actual need.

Pavel Savinov 2018-08-02T07:30:00Z
Categories: CMS, ECM

Talend and Splunk: Aggregate, Analyze and Get Answers from Your Data Integration Jobs

Talend - Wed, 08/01/2018 - 12:26

Log management solutions play a crucial role in an enterprise’s layered security framework— without them, firms have little visibility into the actions and events occurring inside their infrastructures that could either lead to data breaches or signify a security compromise in progress.

Splunk is the “Google for log files” heavyset enterprise tool that was the first log analysis software and has been the market leader ever since. So lots of customers will be interested in seeing how Talend can integrate with their enterprise Splunk and leverage Splunk’s out of the box features.

Splunk captures, indexes, and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations. It has an API that allows for data to be captured in a variety of ways.

Splunk’s core offering collects and analyzes high volumes of machine-generated data. It uses a standard API to connect directly to applications and devices. It was developed in response to the demand for comprehensible and actionable data reporting for executives outside a company’s IT department.

Splunk has several products but in this blog, we will only be working with Splunk Enterprise to aggregate, analyze and get answers from your Talend job logs. I’ll also cover an alternative approach where developers can also log customized events to a specific index using the Splunk Java SDK. Let’s get started!

Intro to Talend Server Log

Let’s start by introducing you to the Talend Log Server. Simply put, this is a logging engine based on Elasticsearch which is developed alongside a data-collection and log-parsing engine called Logstash, and an analytics and visualization platform called Kibana (or ELK).

These technologies are used to streamline the capture and storage of logs from Talend Administration Center, MDM Server, ESB Server and Tasks running through the Job Conductor. It is a tool for managing events and Job logs. Talend supports the basic installation but features like HA and APIs to read/write are beyond Talend scope of supportability.

To understand configuring Talend logging modules with an external Elastic stack please read this article.

Configure Splunk to Monitor Job Logs

Now that you have a good feel for the Talend Server Log, let’s set up Splunk to actually monitor and collect data integration job logs. After you log into your Splunk deployment, the Home page appears. To add data, click Add Data. The Add Data page appears. If your Splunk deployment is a self-service Splunk Cloud deployment, from the system bar, click Settings > Add Data.

The Monitor option lets you monitor one or more files, directories, network streams, scripts, Event Logs (on Windows hosts only), performance metrics, or any other type of machine data that the Splunk Enterprise instance has access to. When you click Monitor, Splunk Web loads a page that starts the monitoring process.

Select a source from the left pane by clicking it once. The page is displayed based on the source you selected. In our case we want to monitor Talend job execution logs, select “Files & Directories”, the page updates with a field to enter a file or directory name and specify how Splunk software should monitor the file or directory. Follow the on-screen prompts to complete the selection of the source object that you want to monitor. Click Next to proceed to the next step in the Add data process.

Creating a Simple Talend Spark Job

To start, log in to your Talend Studio and create a simple job that will read a string via context variable, extract first three characters and displays both actual and extracted string.

Creating Custom Log Events from Talend Spark Job

Now that we’ve gotten everything set up, we’ll want to leverage the Splunk SDK to create custom (based on each flow in the Talend job) events and send it back to Splunk server. A user routine is written to make Splunk calls and register the event to an index. The Splunk SDK jar is set up as a dependency to the user routines so that leverage Splunk SDK methods

Here is how to quickly build the sample Talend Job Below:

  • Splunk configuration is created as context and passed to routine via tJava component
  • Job started and its respective event is logged
  • Employee data is read and its respective event is logged
  • Department data is read and its respective event is logged
  • Employee and Department datasets are joined to form a de-normalized data and its respective event is logged

Switch back to Splunk and search with the index used in the above job – you’ll be able to see events published from job.


Using the exercise and process above, it is clear that Talend can seamlessly connect to Enterprise Splunk and push customized events and complete job log files to Splunk.

The post Talend and Splunk: Aggregate, Analyze and Get Answers from Your Data Integration Jobs appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

Announcing CiviCRM 5.4 Release

CiviCRM - Wed, 08/01/2018 - 11:44
This latest CiviCRM 5.4 release is now ready to download.  RELEASE NOTES: Big thanks to Andrew Hunt from AGH Strategies for putting up together release notes for this version.  The release notes for 5.4 can be accessed here.   SPECIAL THANKS:
Categories: CRM

PrestaShop is one of the first to join new Stripe partner program

PrestaShop - Wed, 08/01/2018 - 03:21
Most people today don’t know that only three percent of GDP is online.
Categories: E-commerce

Monitor c3p0 in Liferay

Liferay - Tue, 07/31/2018 - 20:05
The Problem

The c3p0 connection pool exposes MBeans for JMX tools like jconsole to monitor its run-time states. However, both c3p0 and Liferay intentionally generate random MBean names on each startup. The random name of each data source makes it difficult to setup an independent monitoring tool.

The second problem is, by default Liferay creates several data sources, one used by most Liferay features, one for the counter service and one for analytics. They all use the same portal property prefix "jdbc.default.", making them identical in every way except for their random bean names.

The Solution

A default c3p0 bean name looks like this:


The default behavior sets the value of both  identityToken and name to the same random string. According to c3p0 documentation, identityToken can be removed with a system property. The second part of the solution is explicitly name the data sources created by Liferay.

Remove identityToken

Add the following line to TOMCAT_HOME/bin/

JAVA_OPTS="$JAVA_OPTS" Explicitly Name Data Sources

Add the following lines to LIFERAY_HOME/

jdbc.default.dataSourceName=liferayDataSource counter.jdbc.prefix=jdbc.counter. jdbc.counter.dataSourceName=counterDataSource

These lines name the default data source liferayDataSource, the counter service data source counterDataSource and similarly analyticsDataSource. These names are all arbitrary, so you can chose other desired values.

The second line gives the counter service data source a new property prefix "jdbc.counter." Use this prefix to set counterDataSource properties you wish to be different from the liferayDataSource. Conversely, any properties not set with this prefix will use the value set using the "jdbc.default." prefix.

Now in JMX, c3p0 beans look like this:

  • com.mchange.v2.c3p0:name=analyticsDataSource,type=PooledDataSource
  • com.mchange.v2.c3p0:name=counterDataSource,type=PooledDataSource
  • com.mchange.v2.c3p0:name=liferayDataSource,type=PooledDataSource

Name Other Data Sources

If your Liferay is configured with other data source(s), you must also give each of them a unique name. Otherwise, multiple MBeans will have the same name as liferayDataSource, and only one of them will be detected by JMX tools.

For example, if you have a third data source defined with prefix "", add this line to LIFERAY_HOME/

Then you should see these data sources:

  • com.mchange.v2.c3p0:name=analyticsDataSource,type=PooledDataSource

  • com.mchange.v2.c3p0:name=counterDataSource,type=PooledDataSource

  • com.mchange.v2.c3p0:name=liferayDataSource,type=PooledDataSource

  • com.mchange.v2.c3p0:name=sapDataSource,type=PooledDataSource

What to Watch

In a JMX monitoring tool, for each data source, monitor these 3 properties of its MBean:

  • numConnectionsAllUsers

  • numBusyConnectionsAllUsers

  • numIdleConnectionsAllUsers

An alert should be configured when numBusyConnectionsAllUsers exceeds a certain percentage of maxPoolSize, which forecasts a possible exhaustion of database connections with that data source.


Michael Chen 2018-08-01T01:05:00Z
Categories: CMS, ECM

Dynamic Widget

Liferay - Tue, 07/31/2018 - 15:20

My colleague Vagif proposed a new way to develop a Liferay dynamic widget (the best name I can think of for now) with these pieces:

  1. Use the “Basic Web Content” structure or define a new structure.
  2. For each dynamic widget type, define a new display template.
  3. Create an OSGi service that returns a data model to be used by the display template.
  4. Create a web content article with the above structure/template pair.
  5. Place this web content article onto the page with Web Content Display (OOTB).

All business logics are in the OSGi service. The structure can serve as a preference or parameter holder for the display template. The display template draws most meaningful data from the model returned by the service.

You can build a portlet-less Liferay site with this "Crafter-ish" approach.


A typical Liferay feature has program code spread out among many technologies: service Java code, portlet Java code, template code, template Java code (JSP), language files, property files, XML files, etc. You often find a feature morphs into some JSP files with 100+ lines of Java code, a 2000 line portlet, plus several service builders, all competing for business logic implementations and MVC roles.

A Simpler Pattern

Dynamic widget may serve as a Liferay implementation pattern that simplifies and promotes good programming practices (I think this is better pattern, but some may disagree):

  • Write Java code in and only in service modules (no portlet).
  • Implement all business logic in services.
  • The display template calls a single service to retrieve a data model.
  • The display template then renders the model with almost no business logic.

A Simple Rule:

  • Let the services build the most foolproof model for the template to render it in the most simplistic way.

There is nothing stopping a template from calling multiple services then juggle multiple models to render the view. However, do your self a favor, write a new service method that combines all of them into a single model for the template. You will thank yourself later.

Why Freemarker?

Cannot use JSP may be a down side of dynamic widget, or is it?

Compared to JSP, template languages like Freemarker cannot mix Java code with UI code, so they promote a cleaner separation of view from model and controller. Some may argue that template code getting the model from services is controller-ish. Agree, but beyond that keeping Freemarker code simple is not hard at all, because complex Freemarker code is ugly and painful to write, not to mention tough to debug, log, handle errors, etc.

Pseudo Web Content

Dynamic widget is a Liferay web content, but its "content" is not in its web content structure. The content comes from the services. If you think of a Web Content Display rendering a dynamic widget in a page as calling a function in a program, then its web content structure is like the parameter for that function call.

The widget's web content may be empty, which is similar to calling a function with no parameter. The service that builds the model for the template has everything it needs. In other instances, the structure can have variables used as parameters for calling the service or be used directly by the template just like any other web content.

Search and Preview

Note that variables in the web content structure can be searchable, making them available to Liferay search and Assert Publisher. For example, a dynamic widget may be titled "Annual Sales Figures 2018", which renders a bar chart from a model provided by a service module. Then this widget may be a search hit, along with its pretty chart if you wish. You can't have that if the chart is implemented in a portlet.

Don't forget, you can easily mark the web content not searchable too:

Another convenience of dynamic widget over portlet is preview. Open Liferay Control Panel > (a site) > Web Content. You can preview a widget independent of a page from its Options menu:

Managing Display Templates

For the rest of this topic, lets call the template associated with the web content structure the parent template. The parent template can render the entire widget all by itself, but it's much more likely for the parent template to use other templates with the '<#include />' Freemarker directive.

Here we discuss three ways to develop templates for dynamic widgets.

Liferay Generic Templates

Open Liferay Control Panel > (a site) > Web Content > (corner Options menu) > Templates. Note the first column of table view called "ID". The value of that column is the template ID, also known as template key.

From this Templates view, you can create a template and leave the "Structure" field empty. That creates a generic template. Then in a parent template, you can use a generic template like this:

<#include "${templatesPath}/TEMPLATE_KEY"/>

With this framework, you can implement everything inside Liferay user interface. However, exporting then importing generic templates a LAR file will invalidate all template keys. All parent templates must be manually modified with new keys.

Package Templates In A Module

In the attached, "modules/my-templates" demonstrates packaging Freemarker templates in a OSGi module. The key for parent templates to use templates in this module is the "Web-ContextPath" header:

modules/my-templates/bnd.bnd Bundle-Name: My Templates Bundle Bundle-SymbolicName: my.liferay.templates Bundle-Version: 1.0.0 Web-ContextPath: /my-templates

In a parent template, use the Liferay infix _SERVLET_CONTEXT_ like this:

<#include "my-templates_SERVLET_CONTEXT_/widget/software-project.ftl" />

where before the infix is the "Web-ContextPath" value, and after which is the template path in the module project under "src/main/resources".

The Liferay Freemarker engine allows a template to include sibling templates in the same module using relative path of the current template. For example,

modules/my-templates/src/main/resources/widget/software-project.ftl <h3>software-project.ftl</h3> From parent path: <#include "../common/left-navigation.ftl"/> From child path: <#include "more/right-navigation.ftl"/>

Now the fun part of how Freemarker templates find and call services. First of all, you need to remove Liferay's default restriction on template variables by creating this configuration file as the following:

LIFERAY_HOME/osgi/configs/com.liferay.portal.template.freemarker.configuration.FreeMarkerEngineConfiguration.cfg restrictedVariables=

Then a template can access services in several ways, for example, using the "staticUtil" or "serviceLocator" variables:

modules/my-templates/src/main/resources/widget/software-project.ftl Using restricted variables: <#assign userLocalServiceUtil = staticUtil['com.liferay.portal.kernel.service.UserLocalServiceUtil'] /> ${userLocalServiceUtil.getDefaultUserId(companyId)}, <#assign userLocalService = serviceLocator.findService('com.liferay.portal.kernel.service.UserLocalService') /> ${userLocalService.getDefaultUser(companyId).originalEmailAddress}

Refer to the "Extras" section for a complete list of all Liferay variables available to Freemarker templates.

Package Templates In A Theme

The attached also includes an example theme in "wars/my-theme". Following the folder convention of a theme, templates in the projects are under folder "src/main/webapp/templates". The Gradle build automatically generates a "Web-ContactPath" header with the value of the project folder. Therefore, a parent template can reference a template in this theme as:

<#include "my-theme_SERVLET_CONTEXT_/templates/widget/software-project.ftl" />

Other aspects of templates in a theme are identical to ones in a module.


You can deploy both the module JAR and the theme WAR files by copying them to the Liferay auto deploy folder. Here is what they look like when running:

$ telnet localhost 11311 Trying Connected to localhost. Escape character is '^]'. ____________________________ Welcome to Apache Felix Gogo g! lb my START LEVEL 20 ID|State |Level|Name 759|Active | 10|Liferay My Account Web (1.0.11) 826|Active | 10|Liferay Portal Security AntiSamy (2.0.12) 994|Active | 10|Liferay Site My Sites Web (1.0.8) 1232|Active | 10|My Templates Bundle (1.0.0) 1237|Active | 10|my-theme (7.0.10) g! Extras

A complete list of all Liferay variables available to Freemarker templates, including their implementation class names and snippets of Liferay source code: @Meta.AD( deflt = "serviceLocator|utilLocator|objectUtil|staticFieldGetter|staticUtil", required = false ) public String[] restrictedVariables(); com.liferay.portal.template.TemplateContextHelper.getHelperUtilities(, false) { accountPermission=com.liferay.portal.service.permission.AccountPermissionImpl, arrayUtil=com.liferay.portal.kernel.util.ArrayUtil_IW,, auditRouterUtil=null, browserSniffer=com.liferay.portal.servlet.BrowserSnifferImpl, calendarFactory=com.liferay.portal.util.CalendarFactoryImpl, commonPermission=com.liferay.portal.service.permission.CommonPermissionImpl, dateFormatFactory=com.liferay.portal.util.FastDateFormatFactoryImpl, dateFormats=com.liferay.portal.util.FastDateFormatFactoryImpl, dateTool=May 5, 2018 10:34:58 AM, dateUtil=com.liferay.portal.kernel.util.DateUtil_IW,, expandoColumnLocalService=com.liferay.portlet.expando.service.impl.ExpandoColumnLocalServiceImpl, expandoRowLocalService=com.liferay.portlet.expando.service.impl.ExpandoRowLocalServiceImpl, expandoTableLocalService=com.liferay.portlet.expando.service.impl.ExpandoTableLocalServiceImpl, expandoValueLocalService=com.liferay.portlet.expando.service.impl.ExpandoValueLocalServiceImpl, getterUtil=com.liferay.portal.kernel.util.GetterUtil_IW, groupPermission=com.liferay.portal.service.permission.GroupPermissionImpl, htmlUtil=com.liferay.portal.util.HtmlImpl, httpUtil=com.liferay.portal.util.HttpImpl, imageToken=com.liferay.portal.webserver.WebServerServletTokenImpl, imageToolUtil=com.liferay.portal.image.ImageToolImpl,, jsonFactoryUtil=com.liferay.portal.json.JSONFactoryImpl, languageUtil=com.liferay.portal.language.LanguageImpl, layoutPermission=com.liferay.portal.service.permission.LayoutPermissionImpl,, localeUtil=com.liferay.portal.kernel.util.LocaleUtil, locationPermission=com.liferay.portal.service.permission.OrganizationPermissionImpl,,, organizationPermission=com.liferay.portal.service.permission.OrganizationPermissionImpl, paramUtil=com.liferay.portal.kernel.util.ParamUtil_IW, passwordPolicyPermission=com.liferay.portal.service.permission.PasswordPolicyPermissionImpl, portal=com.liferay.portal.util.PortalImpl, portalPermission=com.liferay.portal.service.permission.PortalPermissionImpl, portalUtil=com.liferay.portal.util.PortalImpl, portletModeFactory=com.liferay.portal.kernel.portlet.PortletModeFactory_IW, portletPermission=com.liferay.portal.service.permission.PortletPermissionImpl, portletProviderAction={ADD=ADD, BROWSE=BROWSE, MANAGE=MANAGE, EDIT=EDIT, PREVIEW=PREVIEW, VIEW=VIEW}, portletURLFactory=com.liferay.portlet.PortletURLFactoryImpl, prefsPropsUtil=com.liferay.portal.util.PrefsPropsImpl, propsUtil=com.liferay.portal.util.PropsImpl, randomizer=com.liferay.portal.kernel.util.Randomizer, rolePermission=com.liferay.portal.service.permission.RolePermissionImpl, saxReaderUtil=com.liferay.portal.xml.SAXReaderImpl, serviceLocator=com.liferay.portal.template.ServiceLocator, sessionClicks=com.liferay.portal.kernel.util.SessionClicks_IW,, staticFieldGetter=com.liferay.portal.kernel.util.StaticFieldGetter, stringUtil=com.liferay.portal.kernel.util.StringUtil_IW, timeZoneUtil=com.liferay.portal.kernel.util.TimeZoneUtil_IW, unicodeFormatter=com.liferay.portal.kernel.util.UnicodeFormatter_IW, unicodeLanguageUtil=com.liferay.portal.language.UnicodeLanguageImpl, userGroupPermission=com.liferay.portal.service.permission.UserGroupPermissionImpl, userPermission=com.liferay.portal.service.permission.UserPermissionImpl, utilLocator=com.liferay.portal.template.UtilLocator, validator=com.liferay.portal.kernel.util.Validator_IW, velocityPortletPreferences=, webServerToken=com.liferay.portal.webserver.WebServerServletTokenImpl, windowStateFactory=com.liferay.portal.kernel.portlet.WindowStateFactory_IW, }


Michael Chen 2018-07-31T20:20:00Z
Categories: CMS, ECM

Fundamentals of Membership Management online training - August 7th

CiviCRM - Tue, 07/31/2018 - 14:11

Join Cividesk on Tuesday, August 7th at 1 pm PT /2 pm MT/ 4 pm ET for this 2-hour online training session on the basics of CiviMember.  

Learn to set up membership types, record, search and report on member data, send membership renewal emails and much more!

Click here for more information and to register/pay for this informative training session. Questions? email Cividesk training manager, Susan Engeman at

Categories: CRM

A Simplified Pattern for Liferay 7 Services

Liferay - Tue, 07/31/2018 - 13:07

This is a simplified OSGi service API and implementation pattern. It follows the traditional Java interface-implementation pattern, in which the programmer is only required to keep the interface class and implementation class in sync. It does not use Liferay 7 service builder.

The attached archive is a fully implemented ORM service based on MyBatis. Unzip it into the modules folder in a Blade created workspace. A script to create the back-end database will be added soon to make this example fully operational.

Implementation Pattern

In this ORM example, there are two top level packages: 'api' and 'impl'. The 'api' and its children packages are to be exported and used by consumers of this service API. The 'impl' and its children packages are for implementation only and should remain private packages of this OSGi module.

Factory Usage Pattern

The 'api.Factory' class is the access point for consumers to get the services they need. A consumer class uses the Factory like this:

import com.acme.orm.api.bean.Order import static com.acme.orm.api.Facotry.getOrderLocalService; class OnlineStore { public Order checkOrder(String orderId) { Order order = getOrderLocalService().getOrderDetailsById(orderId); // Do something else return order; } }

In order to preserve OSGi's runt-time life cycle management of this module (start, stop, install, uninstall), it is important NOT to keep a reference of the service object obtained from the Factory:

// DO NOT DO THIS OrderLocalService myService = getOrderLocalService(); // DO NOT DO THIS OrderLocalService myService = Factory.getOrderLocalService();

The Liferay service builder went to great length to prevent programmers from keeping a reference of the service object by generating and forcing people to use the static methods in the XyzServiceUtil class. It also creates other confusing and irrelevant artifacts: XyzServiceBaseImpl, XyzServiceWrapper and two projects (two jars) for one service.

Instead of making it foolproof with all those complexities, why not just tell programmers, a very intelligent bunch, not to keep references of OSGi service objects. The result is this clean implementation pattern, with no generated artifact, and two easy to understand rules:

  • Keep the API interface class in sync with the implementation class.
  • Do not keep a reference of the service object obtained from the Factory.
Understanding ServiceTracker

When an OSGi module (or bundle) is replaced at run-time due to it being stopped, started, uninstalled or re-installed, the desired effect is that the services provided by that module be replaced as well. ServiceTracker is the OSGi class that keeps track of module life cycle changes. Module life cycle changes are transparent to service consumer code as long as the consumer code always access the service from the ServiceTracker.

OSGi is a component framework running in a JVM instance that exhibits the same run-time behavior as any Java programs. When the consumer code saves a reference to a service object, that service object will live on even when OSGi replaced its module with a new instance. That service object now becomes an orphan and out-dated instance only known to that consumer code. This is the reason for not keeping a reference to the service.

In this implementation pattern, the Factory class retrieves the service object from its corresponding ServiceTracker. The getService() method of the ServiceTracker shields the module's life cycle changes from the consumer code:

@ProviderType public class Factory { private static ServiceTracker<OrderLocalService, OrderLocalService> _OrderLocalService =; public static OrderLocalService getOrderLocalService() { return _OrderLocalService.getService(); } } Local vs. Remote Service

The differences between a Liferay 7.0 local service and remote service are:

  1. The base interface of the API.
  2. Specific annotations for the remote interface.

In the ORM example, OrderLocalService is local service interface:

Local Service API Declaration @ProviderType @Transactional(isolation = Isolation.PORTAL, rollbackFor = { PortalException.class, SystemException.class}) public interface OrderLocalService extends BaseLocalService { }

while OrderService is the remote service interface exposed as RESTful web service:

Remote Service API Declaration @AccessControlled @JSONWebService @OSGiBeanProperties(property = { "", "json.web.service.context.path=Order" }, service = OrderService.class) @ProviderType @Transactional(isolation = Isolation.PORTAL, rollbackFor = { PortalException.class, SystemException.class}) public interface OrderService extends BaseService { public Order getOrderDetailsById(String orderId); }

This RESTful web service can be found in the following catalog under Context Name "acme" (click the drop down box to find "acme" or other context names):


Both the local and remote service implementation classes just implement their corresponding API interfaces. In the ORM example, OrderLocalServiceImpl is the local implementation that does the actual work of mapping to the database. The remote implementation, as shown below, simply calls the local Factory services:

Remote Service Implementation @ProviderType public class OrderServiceImpl implements OrderService { public Order getOrderDetailsById(String orderId) { return Factory.getOrderLocalService().getOrderDetailsById(orderId); } } Development Details

Here are some key files for creating and implementing a service in this pattern.

Eclipse .project and .classpath

These two files in the example archive must be used to start your service project for Eclipse to recognize it as a Gradle project. You can change the project name in the <name> tag of the .project file before importing the project to Eclipse:


You must also create these two folder structures to hold your Java and resource files:


Once imported to Eclipse, be sure to right click on the project and select "Gradle" -> "Refresh Gradle Project". You can also do the same thing with the parent Gradle project created by Liferay Blade.


The two 'org.osgi:org.osgi.*' dependencies are required for OSGi features.

dependencies { compile group: "com.liferay", name: "com.liferay.osgi.util", version: "3.0.3" compile group: "com.liferay.portal", name: "com.liferay.portal.kernel", version: "2.0.0" compile 'org.osgi:org.osgi.core:5.0.0' compile 'org.osgi:org.osgi.annotation:6.0.0' compile group: 'javax.servlet', name: 'servlet-api', version: '2.5' compile group: "org.mybatis", name: "mybatis", version: "3.4.1" compile files('./resources/lib/sqljdbc4.jar') compileOnly group: "com.liferay", name: "com.liferay.journal.api", version: "1.0.0" } bnd.bnd

All packages under 'api' should be exported in the "Export-Package:" setting. The "Liferay-Spring-Context:" setting directs Liferay to load the Spring bean definition in the module-spring.xml file discussed below. "Lifer-Require-SchemaVersion:", "Liferay-Service:" and "Require-Capability:" settings are also required.

Bundle-Version: 1.0.0 Bundle-ClassPath: .,lib/sqljdbc4.jar Export-Package: \ com.acme.orm.api,\ com.acme.orm.api.bean,\ com.acme.orm.api.exception Import-Package: \ !*,\ !microsoft.sql.*,\ !com.sun.jdi.*,\ !net.sf.cglib.proxy.*,\ !org.apache.logging.*,\ * Include-Resource: @mybatis-3.4.1.jar Liferay-Require-SchemaVersion: 1.0.0 Liferay-Service: true Liferay-Spring-Context: META-INF/spring Require-Capability: liferay.extender;filter:="(&(liferay.extender=spring.extender)(version>=2.0)(!(version>=3.0)))" src/main/resources/META-INF/spring/module-spring.xml

For each bean definition, the "class=" value is the implementation class name, and the "id=" value is the interface class name.

<?xml version="1.0"?> <beans xmlns="" xmlns:xsi="" default-destroy-method="destroy" default-init-method="afterPropertiesSet" xsi:schemaLocation=""> <bean class="com.acme.orm.impl.CommunicationMediumLocalServiceImpl" id="com.acme.orm.api.CommunicationMediumLocalService" /> <bean class="com.acme.orm.impl.MessageCenterLocalServiceImpl" id="com.acme.orm.api.MessageCenterLocalService" /> <bean class="com.acme.orm.impl.NSMUserLocalServiceImpl" id="com.acme.orm.api.NSMUserLocalService" /> <bean class="com.acme.orm.impl.OrderLocalServiceImpl" id="com.acme.orm.api.OrderLocalService" /> <bean class="com.acme.orm.impl.OrderServiceImpl" id="com.acme.orm.api.OrderService" /> <bean class="com.acme.orm.impl.RoutingAreaLocalServiceImpl" id="com.acme.orm.api.RoutingAreaLocalService" /> <bean class="com.acme.orm.impl.WebContentArticleLocalServiceImpl" id="com.acme.orm.api.WebContentArticleLocalService" /> </beans>


Michael Chen 2018-07-31T18:07:00Z
Categories: CMS, ECM

How to disable XML Log Files in Liferay 7.x

Liferay - Tue, 07/31/2018 - 11:52

Liferay 7.0 started to produce a pair of log files per day:

ls -l /opt/liferay/home7/logs/ total 3104 -rw-r--r-- 1 michael admin 285201 May 26 13:24 liferay.2016-05-26.log -rw-r--r-- 1 michael admin 898027 May 26 13:24 liferay.2016-05-26.xml -rw-r--r-- 1 michael admin 400811 Aug 19 13:08 liferay.2016-08-19.log -rw-r--r-- 1 michael admin 0 Aug 19 12:26 liferay.2016-08-19.xml

To disable Liferay logging to XML files, create portal-log4j-ext.xml in the following path:

tomcat-8.0.32/webapps/ROOT/WEB-INF/classes/META-INF/portal-log4j-ext.xml <?xml version="1.0"?> <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> <log4j:configuration xmlns:log4j="">       <appender name="XML_FILE" class="org.apache.log4j.ConsoleAppender"/>       <root>         <priority value="INFO" />         <!--appender-ref ref="CONSOLE" /-->         <appender-ref ref="TEXT_FILE" />         <!--appender-ref ref="XML_FILE" /-->     </root> </log4j:configuration>

This file overrides two elements of META-INF/portal-log4j.xml in ROOT/WEB-INF/lib/portal-impl.jar, which stops Liferay from copying log entries to catalina.out and the daily XML log file.

You can also create the extension XML file as a copy of the original in the JAR as the following:

cd tomcat-8.0.32/webapps/ROOT/WEB-INF unzip -p lib/portal-impl.jar META-INF/portal-log4j.xml > classes/META-INF/portal-log4j-ext.xml


Michael Chen 2018-07-31T16:52:00Z
Categories: CMS, ECM

Joomla 3.8.11 Release

Joomla! - Tue, 07/31/2018 - 08:45

Joomla 3.8.11 is now available. This is a bug fix release for the 3.x series of Joomla including over 35 bug fixes and improvements.

Categories: CMS

Global shopping trends and tourism offer new sales opportunities to merchants

PrestaShop - Tue, 07/31/2018 - 03:06
The global market is opening up more and more for merchants in Europe, bringing with it enormous opportunities, provided that merchants are keeping up with current deve
Categories: E-commerce
Syndicate content