AdoptOS

Assistance with Open Source adoption

Open Source News

Gradle Plugin for manage properties files

Liferay - Mon, 10/15/2018 - 13:01

GitHub repositoty: https://github.com/ironcero/devtools-liferay-portal-properties

Summary

This gradle plugin let you manage your properties files on your liferay workspace (version 7.0 and 7.1).

Blade tool initializes this kind of workpace with one folder named configs. There are some folder into configs folder:

  • common
  • dev
  • local
  • prod
  • uat

It's very common that you need to keep different values for the same properties depends on your environment. This plugin try to help you to manage this settup: copying all properties files from one common folder to each environment folder and replacing all properties found in filters file to the correct value.

How to use

First you will need the plugin jar file. You could download latest version from https://github.com/ironcero/devtools-liferay-portal-properties/blob/master/release/portal-properties-1.0.0.jar (Maven Central Version coming soon) or download source code from this github and to compile it. If you download jar file you will need move this to correct path in your local repository (gradle coordenates are devtools.liferay:portal-properties:1.0.0). Else if you download source code and compile it you will need to execute install maven task to install jar file on correct path in your local repository.

After jar file is fetched you will need to set up your liferay workspace. You will need to create two newely folder. You can create these folder in path you want but we recommend created into common folder (in configs folder).

Now you will need to set up this plugin in your build.gradle file. You will need add these line to build.gradle file:

buildscript { dependencies { classpath group: "devtools.liferay", name: "portal-properties", version: "1.0.0" } repositories { mavenLocal() maven { url "https://repository-cdn.liferay.com/nexus/content/groups/public" } } } apply plugin: "devtools-liferay-portal-properties" buildproperties { descFolderPath = 'configs' originFolderPath = 'configs/common/origin' keysFolderPath = 'configs/common/keys' } build.finalizedBy(buildproperties)

In this example we're going to use configs/common/origin folder to keep original properties file with pattern, and configs/common/keys folder to keep different values for properties. In details:

  • Dependencies: Gradle coordenates of DevTools Liferay Portal Properties is devtools.liferay:portal-properties:1.0.0.
  • Repositories: you will need mavenLocal repository because you've moved plugin jar file to your maven local repository.
  • Apply plugin: DevTools Liferay Portal Properties plugin id is devtools-liferay-portal-properties.
  • BuildProperties: In this section we will put all configuration parameters. In 1.0.0 release we have:
    • descFolderPath: Path where properties file will be copied and properties will be replaced.
    • originFolderPath: Location of original properties file (with ${} filter params).
    • keysFolderPath: Location of filter properties file.
  • build.finaluzedBy: With this command we can execute this plugin on build stage and not only on buildproperties.

It's time to add your properties files.

In the example we've created 4 filter file on keysFolderPath folder (configs/common/keys):

  • dev.properties
  • local.properties
  • prod.properties
  • uat.properties The content of these files are very similar (local.properties):
test1=Local

File name (without .properties extension) must be equals to environment folder on descFolderPath folder.

In the example we've created only one properties file on originFolderPath folder (configs/common/origin). But we'ld put more properties files and all of then would be copied and replaced. portal-ext.properties on configs/common/origin:

testKey=testValue test1Key=${test1}

Now you are be able to generated your portal-ext.properties filtered by environment with buildproperties gradle task, or standar build gradle task.

gradle buildproperties gradle build

This is a common log of process:

:buildproperties Build properties task...configs Settings: destination folder path: configs origin folder path: configs/common/origin keys folder path: configs/common/keys Parsing dev environment... Copying C:\dev\workspaces\devtools\liferay\portal-properties-test\liferay-workspace\configs\common\origin\portal-ext.properties to C:\dev\workspaces\devtools\liferay\portal-properties-test\liferay-workspace\configs\dev WARNING: Property not found in file portal-ext.properties on dev folder (${test1}) WARNING: Property not found in file portal-ext.properties on dev folder (${test2}) Parsing local environment... Copying C:\dev\workspaces\devtools\liferay\portal-properties-test\liferay-workspace\configs\common\origin\portal-ext.properties to C:\dev\workspaces\devtools\liferay\portal-properties-test\liferay-workspace\configs\local Parsing prod environment... Copying C:\dev\workspaces\devtools\liferay\portal-properties-test\liferay-workspace\configs\common\origin\portal-ext.properties to C:\dev\workspaces\devtools\liferay\portal-properties-test\liferay-workspace\configs\prod WARNING: Property not found in file portal-ext.properties on prod folder (${test1}) Parsing uat environment... Copying C:\dev\workspaces\devtools\liferay\portal-properties-test\liferay-workspace\configs\common\origin\portal-ext.properties to C:\dev\workspaces\devtools\liferay\portal-properties-test\liferay-workspace\configs\uat WARNING: Property not found in file portal-ext.properties on uat folder (${test1}) BUILD SUCCESSFUL Total time: 0.275 secs

You will see WARNING log when you have some properties on your original properties files and you haven't filter for these properties on your filter properties files.

You could review Liferay Test project in https://github.com/ironcero/devtools-liferay-portal-properties/tree/master/testProject/liferay-workspace

Ignacio Roncero Bazarra 2018-10-15T18:01:00Z
Categories: CMS, ECM

Astrazeneca: Building the Data Platform of the Future

Talend - Mon, 10/15/2018 - 06:28

AstraZeneca plc is a global, science-led biopharmaceutical company that is the world’s seventh-largest pharmaceutical business, with operations in more than 100 countries. The company focuses on the discovery, development, and commercialization of prescription medicines, which are used by millions of patients worldwide.

It’s one of the few companies to span the entire lifecycle of a medicine; from research and development to manufacturing and supply, and the global commercialization of primary care and specialty care medicines.

Beginning in 2013, AstraZeneca was faced with industry disruption and competitive pressure. For business sustainability and growth, AstraZeneca needed to change their product and portfolio strategy.

As the starting point, they needed to transform their core IT and finance functions. Data is at the heart of these transformations. They had a number of IT-related challenges, including inflexible and non-scalable infrastructure; data silos and diverse data models and file sizes within the organization; a lack of enterprise data governance; and infrastructure over-provisioning for required performance.

The company had grown substantially, including through mergers and acquisitions, and had data dispersed throughout the organization in a variety of systems. Additionally, financial data volume fluctuates depending on where they are in the financial cycle and peaks at month-end, quarters or financial year end are common.

In addition to causing inconsistencies in reporting, silos of information prevented the company and its Science and Enabling Unit division from finding insights hiding in unconnected data sources.

For transforming their IT and finance function and accelerating financial reporting, AstraZeneca needed to put in place a modern architecture that could enable a single source of the truth. As part of its solution, AstraZeneca began a move to the cloud, specifically Amazon Web Services (AWS), where it could build a data lake to hold data from a range of source systems, The potential benefits of a cloud-based solution included increased innovation and accelerated time to market, lower costs, and simplified systems.

But the AWS data lake was only part of the answer. The company needed a way to capture the data, and that’s where solutions such as Talend Big Data and Talend Data Quality come into play. AstraZeneca selected Talend for its AWS connectivity, flexibility, and licensing model, and valued its ability to scale rapidly without incurring extra costs.

The Talend technologies are responsible for lifting, shifting, transforming, and delivering data into the cloud, extracting from multiple sources, and then pushing that data into Amazon S3.

Their IT and Business Transformation initiative was successful and has paved the way for Business Transformation initiatives across five business units and they are leveraging this modern data platform for driving new business opportunities.

Attend this session at Talend Connect UK 2018 to learn more about how AstraZeneca transformed its IT and finance functions by developing an event-driven, scalable data-platform to support massive month-end peak activity, leading to financial reporting in half the time and half the cost.

The post Astrazeneca: Building the Data Platform of the Future appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

Elsevier: How to Gain Data Agility in the Cloud

Talend - Fri, 10/12/2018 - 12:50

Presenting at Talend Connect London 2018 is Reed Elsevier (part of RELX Group), a $7 billion data and analytics company with 31,000 employees, serving scientists, lawyers, doctors, and insurance companies among its many clients. The company helps scientists make discoveries, lawyers win cases, doctors save lives, insurance companies offer customers lower prices, and save taxpayers money by preventing fraud.

Standardizing business practices for successful growth

As the business grew over the years, different parts of the organization began buying and deploying integration tools, which created management challenges for central IT. It was a “shadow IT” situation, where individual business departments were implementing their own integrations with their own different tools.

With lack of standardization, integration was handled separately between different units, which made it more difficult for different components of the enterprise to share data. Central IT wanted to bring order to the process and deploy a system that was effective at meeting the company’s needs as well as scalable to keep pace with growth.

Moving to the cloud

One of the essential requirements was that any new solution be a cloud-based offering. Elsevier a few years ago became a “cloud first” company, mandating that any new IT services be delivered via the cloud and nothing be hosted on-premises. It also adopted agile methodologies and a continuous deployment approach, to become as nimble as possible when bringing new products or releases to market.

Elsevier selected Talend as a solution and began using it in 2016. Among the vital selection factors were platform flexibility, alignment with the company’s existing infrastructure, and its ability to generate Java code as output and support microservices and containers.

In their Talend Connect session, Delivering Agile integration platforms, Elsevier will discuss how it got up and running rapidly with Talend despite having a diverse development environment. And, how it’s using Talend, along with Amazon Web Services, to build a data platform for transforming raw data into insight at scale across the business. You’ll learn how Elsevier created a dynamic platform using containers, serverless data processing and continuous integration/continuous development to reach a level of agility and speed.

Agility is among the most significant benefits of their approach using Talend. Elsevier spins up servers as needed and enables groups to independently develop integrations on a common platform without central IT being a bottleneck. Since building the platform, internal demand has far surpassed the company’s expectations—as it is delivering cost savings and insight at a whole new level.

Attend this session to learn more about how you can transform your integration environment.

 

The post Elsevier: How to Gain Data Agility in the Cloud appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

Events online training session - Tuesday, October 16th

CiviCRM - Fri, 10/12/2018 - 09:41

Cividesk will be offering the intermediate level online training, "Organizing Successful Events" designed for current users familiar with CiviEvent on Tuesday, October 16th from 12 pm MT/ 1 pm CT/ 2 pm ET. 

Categories: CRM

Maintenance extension for PrestaShop 1.6

PrestaShop - Fri, 10/12/2018 - 05:12
It’s been a while since we’ve shared any updates concerning PrestaShop 1.6.
Categories: E-commerce

IP Expo Europe 2018 recap: AI and machine learning on display

SnapLogic - Thu, 10/11/2018 - 17:06

Artificial Intelligence (AI) and machine learning (ML) have come to dominate business and IT discussions the world over. From boardrooms to conferences to media headlines, you can’t escape the buzz, the questions, the disruption. And for good reason – more than any other recent development, AI and ML are transformative, era-defining technologies that are fundamentally[...] Read the full article here.

The post IP Expo Europe 2018 recap: AI and machine learning on display appeared first on SnapLogic.

Categories: ETL

Bitwise: Cloud Data Warehouse Modernization – Inside Look at Talend Connect London

Talend - Thu, 10/11/2018 - 09:28

With expectations of business users evolving beyond limitations of traditional BI capabilities, we see a general thrust of organizations developing a cloud-based data strategy that enterprise users can leverage to build better analytics and make better business decisions. While this vision for cloud strategy is fairly straightforward, the journey of identifying and implementing the right technology stack that caters to BI and analytical requirements across the enterprise can create some stumbling blocks if not properly planned from the get-go.

As a data management consulting and services company, Bitwise helps organizations with their modernization efforts. Based on what we see at our customers when helping to consolidate legacy data integration tools to newer platforms, modernize data warehouse architectures or implement enterprise cloud strategy, Talend fits as a key component of a modern data approach that addresses top business drivers and delivers ROI for these efforts.

For this reason, we are very excited to co-present “Modernizing Your Data Warehouse” with Talend at Talend Connect UK in London. If you are exploring cloud as an option to overcome limitations you may be experiencing with your current data warehouse architecture, this session is for you. Our Talend partner is well equipped to address the many challenges with the conventional data warehouse (that will sound all too familiar to you) and walk through the options, innovations, and benefits for moving to cloud in a way that makes sense to the traditional user.

For our part, we aim to show “how” people are moving to cloud by sharing our experiences for building the right business case, identifying the right approach, and putting together the right strategy. Maybe you are considering whether Lift & Shift is the right approach, or if you should do it in one ‘big bang’ or iterate – we’ll share some practical know-how for making these determinations within your organization.

With so many tools and technologies available, how do you know which are the right fit for you? This is where vendor neutral assessment and business case development, as well as ROI assessment associated with the identified business case, becomes essential for getting the migration roadmap and architecture right from the start. We will highlight a real-world example for going from CIO vision to operationalizing cloud assets, with some lessons learned along the way.

Ultimately, our session is geared to help demonstrate that by modernizing your data warehouse in cloud, you not only get the benefits of speed, agility, flexibility, scalability, cost efficiency, etc. – but it puts you in a framework with inherent Data Governance, Self-Service and Machine Learning capabilities (no need to develop these from scratch on your own), which are the cutting-edge areas where you can show ROI for your business stakeholders…and become a data hero.

Bitwise, a Talend Gold Partner for consulting and services, is proud to be a Gold Sponsor of Talend Connect UK. Be sure to visit our booth to get a demo on how we convert ANY ETL (such as Ab Initio, OWB, Informatica, SSIS, DataStage, and PL/SQL) to Talend with maximum possible automation.

About the author:

Ankur Gupta

EVP Worldwide Sales & Marketing, Bitwise

https://www.linkedin.com/in/unamigo/

The post Bitwise: Cloud Data Warehouse Modernization – Inside Look at Talend Connect London appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

New Talend APAC Cloud Data Infrastructure Now Available!

Talend - Wed, 10/10/2018 - 16:37

As businesses in the primary economic hubs in Asia such as Tokyo, Banglore, Sydney and Singapore are growing at a historical level, they are moving to the cloud like never before. For those companies, their first and foremost priority is to fully leverage the value of their data while meeting strict local data residency, governance, and privacy requirements. Therefore, keeping data in a cloud data center that’s on the other side of the globe simply won’t be enough.

That’s why Talend is launching a new cloud data infrastructure in Japan, in addition to its US data center and the EU data center across Frankfurt and Dublin, in a secure and highly scalable Amazon Web Services (AWS) environment, to allow APAC customers to get cloud data integration and data management services closer to where the data is stored. This is most beneficial to local enterprise businesses and foreign companies who have plans to open up offices in the local region.

There are several benefits Talend Cloud customers can expect from this launch.

Accelerating Enterprise Cloud Adoption

Whether your cloud-first strategy is about modernizing legacy IT infrastructure, leveraging a hybrid cloud architecture, or building a multiple cloud platform, Talend new APAC cloud data infrastructure will allow your transition to the cloud become more seamless. With a Talend Cloud instance independently available in APAC, companies can build a cloud data lake or a cloud data warehouse for faster, more scalable and more agile analytics with more ease.

More Robust Performance

For customers who are using Talend Cloud services in the Asia Pacific, this new cloud data infrastructure will lead to faster extract, transform and load time despite of the data volume. Additionally, it will boost performance for customers using AWS services such as Amazon EMR, Amazon Redshift, Amazon Aurora and Amazon DynamoDB.

Increased Data Security with Proximity

Maintaining data within the local region means the data do not have to make a long trip outside of the immediate area, which can reduce the risk of data security breaches at rest, in transit,  and in use and ease companies’ worries about security measures.

Reduced Compliance and Operational Risks

Because the new data infrastructure offers an instance of Talend Cloud that is deployed independently from the US or the EU, companies can maintain higher standards regarding their data stewards, data privacy, and operational best practices.

For Japan customers, they are likely to be better compliant with Japan’s stringent data privacy and security standards. In the case of industry and government regulation adjustments, Talend Cloud customers would still be able to maintain flexibility and agility to keep up with the changes.

If you are a Talend customer, you will soon have the opportunity to migrate your site to the new APAC data center. Log in or contact your account manager for more information.

Not a current Talend Cloud customers? Test drive Talend Cloud for 30 days free of charge or learn how Talend Cloud can help you connect your data from 900+ data sources to deliver big data cloud analytics instantly.

 

 

 

The post New Talend APAC Cloud Data Infrastructure Now Available! appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

Caldera Forms CiviCRM (CFC) for Donations and Membership

CiviCRM - Wed, 10/10/2018 - 08:19

Tadpole has been working with Andrei Mondoc, the developer of CFC, to extend the availability of processors to support memberships. Since memberships typically use Price Sets, this work also allows for Donations that use Price Sets and Line Items within a contribution page.

Categories: CRM

Five questions to ask about data lakes

SnapLogic - Tue, 10/09/2018 - 18:15

Data is increasingly being recognized as the corporate currency of the digital age. Companies want to leverage data to achieve deeper insights leading to competitive advantage over their peers. According to IDC projections, total worldwide data will surge to 163 zettabytes (ZB) by 2025, an increase of 10x the amount of what exists today. The[...] Read the full article here.

The post Five questions to ask about data lakes appeared first on SnapLogic.

Categories: ETL

5 Questions to Ask When Building a Cloud Data Lake Strategy

Talend - Tue, 10/09/2018 - 15:41

In my last blog post, I shared some thoughts on the common pitfalls when building a data lake. As the movement to the cloud gets more and more common, I’d like to further discuss some of the best practices when building a cloud data lake strategy. When going beyond the scope of integration tools or platforms for your cloud data lake, here are 5 questions to ask, that can be used as a checklist:

1. Does your Cloud Data Lake strategy include a Cloud Data Warehouse?

As many differences as there are between the two, people often times compare the two types of technology approaches. Data warehouses being the centralization of structured data, and Data Lakes often times being the holy grail of all types of data. (You can read more about the two approaches here.)

Not to confuse the two, as these technology approaches should actually be brought together. You will need a data lake to accommodate all types of data that your business deal with today, make it structured, semi-structured or unstructured, on-premise or in the cloud, or those newer types of data such as IoT data. The data lake often time has a landing zone and staging zone for raw data – data at this stage are not yet consumable, but you may want to keep them for future discovery or data science projects. On the other hand, a cloud data warehouse will be in the picture after data is cleansed, mapped and transformed, so that it is more consumable for business analysts to access and make the use of data for reporting or other analytical use. Data at this stage is often time highly processed to adjust to the data warehouse.

If your approach currently only works with a cloud data warehouse, then often time you are losing raw and some formats of data already, it is not so helpful for any prescriptive or advanced analytics projects, or machine learning and AI initiatives as some meanings within the data is already lost. Vice versa, if you don’t have a data warehouse alongside with your data lake strategy, you will end up with a data swamp where all data is kept with no structure, and not consumable by analysts.

From the integration perspective, make sure your integration tool work with both data lake and data warehouse technologies, which will lead us to the next question. 

Download Data Lakes: Purposes, Practices, Patterns, and Platforms now.
Download Now

2. Does your integration tool have ETL & ELT?

As much as you may know about ETL in your current on-premises data warehouse, moving it to the cloud is a different story, not to mention in a cloud data lake context. Where and how data is processed really depends on what you need for your business.

Similar to what we described in the first question, sometimes you need to keep more of the raw nature of the data, and other times you need more processing. This would require your integration tool to cope with both ETL and ELT capabilities, where the data transformation can be handled either before the data is loaded to your final target, e.g. a cloud data warehouse, or after data is landed there. ELT is more often leveraged when the speed of data ingestion is key to your project, or when you want to keep more intel about your data. Typically, cloud data lakes have a raw data store, then a refined (or transformed) data store. Data scientists, for example, prefer to access the raw data, whereas business users would like the normalized data for business intelligence.

Another use of ELT refers to the massive parallel processing capabilities coming with big data technologies such as Spark and Flink. If your use case requires such a strong processing power, then ELT is a better choice where the processing has more scalability.

3. Can your cloud data lake handle both simple ETL tasks and complex big data ones?

This may look like an obvious question but when you ask about this question, put yourself in the users’ shoes and really think through if your choice of tool can meet both requirements.

Not all of your data lake usage will be complex ones that require advanced processing and transformation, many of them can be simple activities such as ingesting new data into the data lake. Often times, the tasks go beyond the data engineering or IT team as well. So ideally the tool of your choice should be able to handle simple tasks fast and easy, but also can scale to the complexity to meet the requirements of advanced use cases.  Building a data lake strategy that can cope with both can help you make your data lake more consumable and practical for various types of users for different purposes.

4. How about batch and streaming needs?

You may think your current architecture and technology stack is good enough, and your business is not really in the Netflix business where streaming is a necessity. Get it? Well think again.

Streaming data has become a part of our everyday lives whether you realize it or not. The “Me” culture has put everything at the moment of now. If your business is on social media, you are in streaming. If IoT and sensor is the next growth market for your business, you are in streaming. If you have a website for customer interaction, you are in streaming. In IDC’s 2018 Data Integration and Integrity End User Survey, 93% of the respondents indicate the plan to use streaming technology by 2020. Real-time and streaming analytics have become a must for modern businesses today to create that competitive edge. So, this naturally raises the questions: can your data lake handle both your batch and streaming needs? Do you have the technology and people to work with streaming, which is fundamentally different from typical batch needs?

Streaming data is particularly challenging to handle because it is continuously generated by an array of sources and devices as well as being delivered in a wide variety of formats.

One prime example of just how complicated streaming data can be comes from the Internet of Things (IoT). With IoT devices, the data is always on; there is no start and no stop, it just keeps flowing. A typical batch processing approach doesn’t work with IoT data because of the continuous stream and the variety of data types it encompasses.

So make sure your data lake strategy and data integration layer can be agile enough to work with both use cases.

You can find more tips on streaming data, here.

5. Can your data lake strategy help cultivate a collaborative culture?

Last but not least, collaboration.

It may take one person to implement the technology, but it will take a whole village to implement it successfully. The only way to make sure your data lake is a success is to have people use it, improving the workflow one way or another.

In a smaller scope, the workflow in your data lake should be able to be reused and leveraged among data engineers. Less recreation will be needed, and operationalization can be much faster. In a bigger scope, the data lake approach can help improve the collaboration between IT and business teams. For example, your business teams are the experts of their data and they know the meaning and the context of data better than anyone else. Data quality can be much improved if the business team can work on the data for business rule transformations, while IT still governs that activity. Defining such a line with governance in place is a delicate work and no easy task. But you may think through your data lake approach, whether it’s governed but open at the same time to encourage not only final consume /usage of the data, but the improvement of data quality in the process, and be recycled to be available to a broader organization.

To summarize, there we go the 5 questions I would recommend asking when thinking about building a cloud data lake strategy. By no means are these the only questions you should think, but hopefully it initiates some thinking outside of your typical technical checklist. 

The post 5 Questions to Ask When Building a Cloud Data Lake Strategy appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

How to Implement a Job Metadata Framework using Talend

Talend - Tue, 10/09/2018 - 14:39

Today, data integration projects are not just about moving data from point A to point B, there is much more to it. The ever-growing volumes of data, the speed at which the data changes presents a lot of challenges in managing the end-to-end data integration process. In order to address these challenges, it is paramount to track the data-journey from source to target in terms of start and end timestamps, job status, business area, subject area, and the individuals responsible for a specific job. In other words, metadata is becoming a major player in data workflows. In this blog, I want to review how to implement a job metadata framework using Talend. Let’s get started!

Metadata Framework: What You Need to Know

The centralized management and monitoring of this job metadata are crucial to data management teams. An efficient and flexible job metadata framework architecture requires a number of things. Namely, a metadata-driven model and job metadata.

A typical Talend Data Integration job performs the following tasks for extracting the data from source systems and loading them into target systems.

  1. Extracting data from source systems
  2. Transforming the data involves:
    • Cleansing source attributes
    • Applying business rules
    • Data Quality
    • Filtering, Sorting, and Deduplication
    • Data aggregations
  3. Loading the data into a target systems
  4. Monitoring, Logging, and Tracking the ETL process

Figure 1: ETL process

Over the past few years, the job metadata has evolved to become an essential component of any data integration project. What happens when you don’t have job metadata in your data integration jobs? It may lead to incorrect ETL statistics and logging as well as difficult to handle errors occurred during the data integration process. A successful Talend Data Integration project depends on how well the job metadata framework is integrated with the enterprise data management process.

Job Metadata Framework

The job metadata framework is a meta-data driven model that integrates well with Talend product suite. Talend provides a set of components for capturing the statistics and logging information during the flight of the data integration process.

Remember, the primary objective of this blog is to provide an efficient way to manage the ETL operations with a customizable framework. The framework includes the Job management data model and the Talend components that support the framework.

Figure 2: Job metadata model

Primarily, the Job Metadata Framework model includes:

  • Job Master
  • Job Run Details
  • Job Run Log
  • File Tracker
  • Database High Water Mark Tracker for extracting the incremental changes

This framework is designed to allow the production support to monitor the job cycle refresh and look for the issues relating to job failure and any discrepancies while processing the data loads. Let’s go through each of piece of the framework step-by-step.

Talend Jobs

Talend_Jobs is a Job Master Repository table that manages the inventory of all the jobs in the Data Integration domain.

Attribute

Description

JobID

Unique Identifier to identify a specific job

JobName

Job Name is the name of the job as per the naming convention (<type>_<subject area>_<table_name>_<target_destination>

BusinessAreaName

Business Unit / Department or Application Area

JobAuthorDomainID

Job author Information

Notes

Additional Information related to the job

LastUpdateDate

The last updated date

Talend Job Run Details

Talend_Job_Run_Details registers every run of a job and its sub jobs with statistics and run details such as job status, start time, end time, and total duration of main job and sub jobs.

Attribute

Description

ID

Unique Identifier to identify a specific job run

BusinessAreaName

Business Unit / Department or Application Area

JobAuthorDomainID

Job author Information

JobID

Unique Identifier to identify a specific job

JobName

Job Name is the name of the job as per the naming convention (<type>_<subject area>_<table_name>_<target_destination>

SubJobID

Unique Identifier to identify a specific sub job

SubJobName

Sub Job Name is the name of the sub job as per the naming convention (<type>_<subject area>_<table_name>_<target_destination>

JobStartDate

Main Job Start Timestamp

JobEndDate

Main Job End Timestamp

JobRunTimeMinutes

Main Job total job execution duration

SubJobStartDate

Sub Job Start Timestamp

SubJobEndDate

Sub Job End Timestamp

SubJobRunTimeMinutes

Sub Job total job execution duration

SubJobStatus

Sub Job Status (Pending / Complete)

JobStatus

Main Job Status (Pending / Complete)

LastUpdateDate

The last updated date

Talend Job Run Log

Talend_Job_Run_Log logs all the errors occurred during particular job execution. Talend_Job_Run_Log extracts the details from the Talend components specially designed for catching logs (tLogCatcher) and statistics (tStatCacher).

Figure 3: Error logging and Statistics

The tLogCatcher component in Talend operates as a log function triggered during the process by one of these components: Java exceptions, tDie or tWarn. In order catch exceptions coming from the job, tCatch function needs to be enabled on all the components.

The tStatCatcher component gathers the job processing metadata at the job level.

Attribute

Description

runID

Unique Identifier to identify a specific job run

JobID

Unique Identifier to identify a specific job

Moment

The time when the message is caught

Pid

The Process ID of the Job

parent_pid

The Parent process ID

root_pid

The root process ID

system_pid

The system process ID

project

The name of the project

Job

The name of the Job

job_repository_id

The ID of the Job file stored in the repository

job_version

The version of the current Job

context

The Name of the current context

priority

The priority sequence

Origin

The name of the component if any

message_type

Begin or End

message

The error message generated by the component when an error occurs. This is an After variable. This variable functions only if the Die on error checkbox is cleared.

Code

 

duration

Time for the execution of a Job or a component with the tStatCaher Statistics check box selected

Count

Record counts

Reference

Job references

Thresholds

Log thresholds for managing error handling workflows

Talend High Water Marker Tracker

Talend_HWM_Tracker helps in processing delta and incremental changes of a particular table. The High Water Tracker is helpful when the “Change Data Capture” is not enabled and the changes are extracted based on specific conditions such as “last_updated_date_time” or ‘revision_date_time.” In some cases, the High Water Mark relates to the highest sequence number when the records are processed based on the sequence number.

 Attribute

Description

Id

Unique Identifier to identify a specific source table

jobID

Unique Identifier to identify a specific job

job_name

The name of the Job

table_name

The name of the source table

environment

The source table environment

database_type

The source table database type

hwm_datetime

High Water Field (Datetime)

hwm_integer

High Water Field (Number)

hwm_Sql

High Water SQL Statement

Talend File Tracker

Talend_File_Tracker registers all the transactions related to file processing. The transaction details include source file location, destination location, file name pattern, file name suffix, and the name of the last file processed.

Attribute

Description

Id

Unique Identifier to identify a specific source file

jobID

Unique Identifier to identify a specific job

job_name

The name of the Job

environment

The file server environment

file_name_pattern

The file name pattern

file_input_location

The source file location

file_destination_location

The target file location

file_suffix

The file suffix

latest_file_name

The name of the last file processed for a specific file

override_flag

The override flag to re-process the file with the same name

update_datetime

The last updated date

Conclusion

This brings to the end of the implementing Job metadata framework using Talend. The following are key takeaways from this blog:

  1. The need and the importance of Job metadata framework
  2. The data model to support the framework
  3. The customizable data model to support different types of job patterns.

As always – let me know if you have any questions below and happy connecting!

The post How to Implement a Job Metadata Framework using Talend appeared first on Talend Real-Time Open Source Data Integration Software.

Categories: ETL

Giving Back: Chase The Music

CiviCRM - Tue, 10/09/2018 - 13:12

Chase the Music gives children battling critical conditions love, hope, strength and joy. We do this with original music - composed and performed just for them.

The impact of Chase the Music programs begins with the child. The children (that we write and perform for) are undergoing situations that no child should.

Categories: CRM

Joomla 3.8.13 Release

Joomla! - Tue, 10/09/2018 - 08:45

Joomla 3.8.13 is now available. This is a security release for the 3.x series of Joomla which addresses 5 security vulnerabilities.

Categories: CMS

Liferay Screens meets React Native, the sequel

Liferay - Mon, 10/08/2018 - 11:01

First of all, for those of you who don't know about Liferay Screens. Liferay Screens is a component library based on components called Screenlets. A screenlet is a visual component that you insert into your native app to leverage Liferay’s content and services, allowing you to create complex native applications for iOS and Android very fast. Awesome, isn’t it?

BUT, Do you need to create the SAME application for iOS and Android, with the SAME features twice? Ok, with screenlets it does not take too much time because most of the boring logic is encapsulated inside the screenlet and you only need to connect the dots. But it could be fantastic to have only one project and share the code of the two platforms.

How can we make this possible? Have you heard about React Native?

Goals

As you may know, React Native is a framework that allows you to create native applications (Android and iOS) in javascript using React. This avoids the necessity of having to maintain two different codebases, one per platform. It’s based on components, so the screenlets concept suits very well in React.

Long time ago, when ReactNative was released, we made a first proof of concept with some of the screenlets available at that moment. Now, we have came back to this idea and we have made another proof of concept. This one will feature all our brand new and more complex screenlets and, yes, Android is supported too.

With this prototype we aim to provide a solution to make mobile apps development faster (yeah, even more!) with React Native, so we could use the screenlets the same way you would use any react native component, like a Button component. Great! Do you want to see how it works? Take a look of the next video, it shows you how to use our library in React Native.

As you can see in the video, the use of screenlets from React Native are very easy. You only have to instantiate the screenlet that you want to use, give it a style with height and width because otherwise the screenlet will not show; and if you consider it appropriate handle the events that the screenlet will send.

To handle an event you have to specify a callback function that manage the mentioned event. E.g., in LoginScreenlet you can handle the event onLoginSuccess to handle when the user log in correctly.

Of course, the attributes (known as props in React) of the screenlets depends on the screenlet that will use, so some screenlets will have required attributes, e.g., the UserPortraitScreenlet needs the userId attribute.
To use all of this functionality in your react project, you have to configure your project following the steps of this video. Also, in the project’s README you can find a description of main steps to configure your react native project.

What is the status of the project?

For now this is a prototype. Even so,  ALL screenlets are available in React Native. In total, we have 21 screenlets in Android and 22 in iOS (the fileDisplayScreenlet is only available from iOS). To play with them, we recommend use the most common screenlets, like ImageGalleryScreenlet which show an image gallery, the UserPortraitScreenlet, the CommentListScreenlet to show a comments list of an asset and, of course, the LoginScreenlet, but you can use whatever you want.
So you can explore and tinker with them. Here you have the project.

How it works ?

We don’t want to bore you with technical details. Basically explained, we made a bridge, we built one side of the bridge in the native part, and the other side in the React Native part so it allows communication between them and render the screenlets.

What now?

Well, now it depends on you. You have the project to play with. We are open to suggestions and feedback. Honestly, we are very happy with the result for now.

Thanks for reading.

Luismi.

Luis Miguel Barco 2018-10-08T16:01:00Z
Categories: CMS, ECM

6 tips for online shops on how to use an email marketing tool correctly

PrestaShop - Mon, 10/08/2018 - 04:39
Some online shops are of the opinion that email marketing tools mean too much effort and too little profit. Both can easily be refuted.
Categories: E-commerce

Listing out context variables 

Liferay - Mon, 10/08/2018 - 04:21
What's Context Contributor?

While developing a theme I was thinking how can I know all the variables that I can access in my free marker templates? Won't it be great if I can write a utility program which can list out all the variables and objects that can be accessed from our theme or other templates files? Context Contributor concept can be used for this purpose. Actually using Context Contributor we can write a piece of code to inject contextual information which can be reused in various template files. If you are not aware of context contributor please visit my article http://proliferay.com/context-contributor-concept-in-liferay-7dxp/ . 

How to create context contributor?

Using the Liferay IDE we can create the project structure. 

 

The Code: 

Our context contributor class has to implements TemplateContextContributor interface and we need to implement the below method. 

    public void prepare(Map<String, Object> contextObjects, HttpServletRequest request) {
    //The fun part is here 
    }

If we see the above code, the first parameter contextObjectcts is the map which contains all the contextual information as key-value pairs. We can iterate all the items of the map and write it to a file. Here is the complete code of the method.  This method writes a file in my D drive with a file name all-variables.html.Of course, you can change it the way you want. 

@Override
    public void prepare(
        Map<String, Object> contextObjects, HttpServletRequest request) {
        PrintWriter writer;
        try {
            writer = new PrintWriter("D:\\all-variables.html", "UTF-8");
            StringBuffer stb = new StringBuffer();
            stb.append("<html>");
            stb.append("<body>");
            stb.append("<table border=\"1\">");
            stb.append("<tr>");
            stb.append("<th>Variable Name</th>");
            stb.append("<th>Variable Value</th>");
            stb.append("</tr>");

            for (Map.Entry<String, Object> entry : contextObjects.entrySet()) {
                stb.append("<tr>");
                stb.append("<td>"+entry.getKey()+"</td>");
                stb.append("<td>"+entry.getValue()+"</td>");
                
               // System.out.println(entry.getKey() +" = =  "+ entry.getValue());
                stb.append("</tr>");
            }
            
            stb.append("</table>");
            stb.append("</body>");
            stb.append("</html>");
            
            writer.println(stb.toString());
            writer.close();
        } catch (FileNotFoundException | UnsupportedEncodingException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }    
    }

You just deploy the module and access any pages. This code will be executed and your file is ready. Now you have all the contextual information which can be used the theme development as well as writing ADT. 

The output of the code:

 

Have fun... Hamidul Islam 2018-10-08T09:21:00Z
Categories: CMS, ECM

Contact Management online training for New CiviCRM Users - October 10th

CiviCRM - Sun, 10/07/2018 - 08:55

If you are new to CiviCRM, you should participate in this 2-hour online training taught by Cividesk on Wednesday, October 10th at 9 am MT/ 10 am CT/ 11 am ET.

The Fundamentals of Contact Management focuses on best practices for managing contacts and the relationships between them. We also cover searching, activities, emailing, merging duplicates, groups and tags, and creating an activity report. 

Categories: CRM

Why I'm Flying South to LSNA2018

Liferay - Sat, 10/06/2018 - 20:10
or, How to blow a Saturday evening writing a blog post just because  the kids don't want to hang out with you

 

Here are the 5 reasons I am flying down tomorrow evening.

I've been to LSNA twice before, in 2015 and 2016. I remember the energy and the ambience. Some of the topics deserve waaaay more than the 30 minutes or hour that is allocated to them, but then the presentations are designed to leave you with just enough to take a deep dive on whichever topic interests you, and on that front, they absolutely deliver. So, here's to more of that.

Unconference. I've never attended one of these before, but the prospect has me interested. I mean, it looks like the attendees get to carve out the day's agenda. I'll be bringing my list of topics fwiw. Something tells me there'll be enough knowledge sharing to go all around. 

Speed consulting. I hope I get to reserve a spot. I have a half-baked design approach around a SAML-authentication requirement using the SAML connector plugin - just a lot of holes that need plugging. Hoping a 20-minute session will help clear things up for me.

Agenda topics: As always, great spread! Here's the top 5 items on my radar at this time:

  • Search (Oct 9, 10:35)
  • Securing your APIs for Headless Use (Oct 10, 11:00)
  • Extending Liferay with OSGI (Oct 9, 4:30)
  • Liferay Analytics Cloud (Oct 10, 10:20)
  • Building Amazing Themes (Oct 9, 3:50)

Food. I have on my to-do list to eat a bowl of authentic étouffée. I will have to seek out the best place for this.

Javeed Chida 2018-10-07T01:10:00Z
Categories: CMS, ECM

Member Only Event - CiviCRM Extension

CiviCRM - Fri, 10/05/2018 - 10:03
  • Do you want to restrict registration to certain events to logged in members only but still have other events open to the public? If the answer to the above question is yes, then this extension is for you. This extension allows you to set a flag to any event so registration is restricted to those that have a current membership.
     

Categories: CRM
Syndicate content