You are not logged in.

Palo improvements in Kettle 4.3

Dear visitor, welcome to Palo Community Forum. If this is your first visit here, please read the Help. It explains in detail how this page works. To use all features of this page, you should consider registering. Please use the registration form, to register here or read more information about the registration process. If you are already registered, please login here.

pvanderm

Intermediate

  • "pvanderm" is male
  • "pvanderm" started this thread

Posts: 13

Date of registration: Apr 20th 2010

  • Send private message

1

Monday, September 19th 2011, 2:04am

Palo improvements in Kettle 4.3

Hi Guys

I recently made some big improvements to the Palo plugin in Kettle that will be released with Kettle 4.3.

For those who don't know the history, the plugin was maintained separately to Kettle on SourceForge, but has been merged into Kettle some time ago. The plugin is now maintained by the Kettle community and is included with a standard Kettle installation. You still need to copy the jpalo.jar file to the libext directory since the jpalo GPL licence stops it from being packaged with Kettle.

Improvements include:
1. New Cube Delete Step.
2. New Cube Create Step.
3. Support to specify the consolidation factor when creating dimension elements.
4. Support for the dimension output to return base elements only.
5. Support to recreate a dimension (because a delete all can take forever).
6. Speed improvements on the Cell output by using a caching mechanism and batching updates.
7. Speed improvements on the Dimension output by using a caching mechanism.

A couple of bugs were also fixed.

It has been tested using Palo 3.2, 3.1 and 2.5.

For a full list and details you can go to jira.pentaho.com and search for "palo". You can optionally also filter on "Pentaho Data Integration" to exclude stuff from other projects.

I need some testers to test if the changes I made work and have the desired results. If you are using kettle at the moment, please give it a go. You should appreciate the speed improvements if you are loading a decent sized dataset.

The new Kettle version 4.2 has just been released. No date has been set on 4.3, so you will need to download a CI version from http://ci.pentaho.com/job/Kettle/

If there are any bugs or functions you would like to see in Kettle, please log it on jira.pentaho.com

Kind regards,
Pieter
Kind Regards,
Pieter van der Merwe
De Bortoli Wines

tish1

Sage

Posts: 777

Date of registration: Jul 13th 2009

Location: Vienna / Austria

Occupation: Senior Consultant @ Vector SW DV GmbH

  • Send private message

2

Tuesday, September 20th 2011, 10:14pm

Hi,

one of the bottle-necks when testing Kettle with Palo used to be the performance and the functionality of the Cube Input Step. Did you change anything there?
Btw. Jedox reimplemented the Palo Java API - which is no longer jPalo.jar but PalojLib.jar. It's already available in Palo CE 3.2. During initial testing we were able to observe huge performance benefits.

Regards.
Robert Tischler
Senior Consultant Vector SW DV GmbH

pvanderm

Intermediate

  • "pvanderm" is male
  • "pvanderm" started this thread

Posts: 13

Date of registration: Apr 20th 2010

  • Send private message

3

Sunday, September 25th 2011, 1:37pm

Hi

I didn't change anything on the Cube Input Step as we don't use it much. We use the plugin mostly to load Palo cubes, hence most of the enhancements were made there. I may have a look at it in the next couple of weeks to see if there is something that can be done.

Thanks for the update on the new PalojLib.jar. I wasn't aware of that! I'll definitely look at replacing the old jpalo.jar with the new PalojLib.jar. Hopefully that also helps the Cube Input Step speed.
Kind Regards,
Pieter van der Merwe
De Bortoli Wines

tish1

Sage

Posts: 777

Date of registration: Jul 13th 2009

Location: Vienna / Austria

Occupation: Senior Consultant @ Vector SW DV GmbH

  • Send private message

4

Monday, September 26th 2011, 10:02am

Hi,

I was told that the new library should also be capable to load data to Palo in parallel loads.

Regards.
Robert Tischler
Senior Consultant Vector SW DV GmbH

pvanderm

Intermediate

  • "pvanderm" is male
  • "pvanderm" started this thread

Posts: 13

Date of registration: Apr 20th 2010

  • Send private message

5

Tuesday, September 27th 2011, 2:37pm

I'm having a look at the speed of the cube input and the new API. The cube input does seem slow at around 200 rows/sec (on my old laptop) and the new API does seem to have a cube exporter class that can maybe speed it up?

But having a look at the API, there isn't much help around and with obfuscated class signatures such as: CellsExporter com.jedox.palojlib.interfaces.ICube.getCellsExporter(IElement[][] arg0, ExportType arg1, int arg2, boolean arg3, boolean arg4, boolean arg5) throws PaloException with no javadoc it doesn't help either Converting existing code from jpalo to the new api doesn't seem that difficult, but any new functionality is difficult without some form of documentation.

Do you know if there is any documentation for the API and what the licencing is? I also couldn't find any source for it on the svn repository. I'm reluctant to look at the ELT code since Kettle is LGPL and I won't be able to use any of the ETL code.

Thanks for all the info so far!
Kind Regards,
Pieter van der Merwe
De Bortoli Wines

tish1

Sage

Posts: 777

Date of registration: Jul 13th 2009

Location: Vienna / Austria

Occupation: Senior Consultant @ Vector SW DV GmbH

  • Send private message

6

Wednesday, September 28th 2011, 11:54am

Hi,

if you want a few code samples, please send me a PM with your email address.

I'm no expert concerning licensing, but Talend directly uses Palo's Restful HTTP API without an intermediate like "jPalo". That might be a way to go for ...

Regards.
Robert Tischler
Senior Consultant Vector SW DV GmbH

v_malicevic

Palo Team

  • "v_malicevic" is male

Posts: 455

Date of registration: Oct 26th 2005

Location: Germany

  • Send private message

7

Wednesday, September 28th 2011, 3:29pm

New java library is currently in rampu-up phase. API side of it may change by the time we officially introduce it in 3.3. That is why it is only used internaly and not documented for the outside. Feel free to use it but with caution.
Mit freundlichen Gruessen/ With kind Regards / Meilleures salutations

Vladislav Malicevic
Head of Research and Development

Jedox AG

pvanderm

Intermediate

  • "pvanderm" is male
  • "pvanderm" started this thread

Posts: 13

Date of registration: Apr 20th 2010

  • Send private message

8

Wednesday, September 28th 2011, 3:34pm

Thanks for the update.

I'll give it a try and if there are huge benefits from using the new API, I may be useful to use it now. If not, I'll wait for the official release in 3.3.
Kind Regards,
Pieter van der Merwe
De Bortoli Wines

DMurray3

Trainee

  • "DMurray3" is male

Posts: 4

Date of registration: Aug 10th 2009

Location: Quito, Ecuador

Occupation: BI Consultant

  • Send private message

9

Friday, September 21st 2012, 4:15am

Connecting Pentaho Kettle 4.2 / 4.3 to Palo

Dear Robert, I am a newbie with PaloBI and I am trying to load data to a DB in Palo, using Pentaho Kettle 4.2 and even 4.3 (of course on two different machines).

Based on a portion of your answer to another post:

Quoted from "tish1"

Btw. Jedox reimplemented the Palo Java API - which is no longer jPalo.jar but PalojLib.jar. It's already available in Palo CE 3.2. During initial testing we were able to observe huge performance benefits.
I have copied the "palojlib-1.0.35.jar" found in C:\Program Files (x86)\Jedox\Palo Suite\tomcat\webapps\etlserver\WEB-INF\lib to Kettle´s \libext folder, and when testing the connection, I still cannot connect to Palo's localhost. I just recently installed PaloBi Suite 3.2.0 (Server version 3.2.12227).

I have also reviewed http://jira.pentaho.com/browse/PDI-6852 as well as the link to http://palo.svn.sourceforge.net/viewvc/p…/Java/palojlib/).


Can you advise what the problem could? Must I must re-compile the palojlib from Palo´s source?

Many thanks and kind regards.

DMurray3
Kind regards, DMurray3

tish1

Sage

Posts: 777

Date of registration: Jul 13th 2009

Location: Vienna / Austria

Occupation: Senior Consultant @ Vector SW DV GmbH

  • Send private message

10

Friday, September 21st 2012, 1:44pm

Hi,

do you have Palo and Kettle on the same machine? What's in your palo.ini after "http"?
Do you recieve any error messages?

Regards.
Robert Tischler
Senior Consultant Vector SW DV GmbH

DMurray3

Trainee

  • "DMurray3" is male

Posts: 4

Date of registration: Aug 10th 2009

Location: Quito, Ecuador

Occupation: BI Consultant

  • Send private message

11

Friday, September 21st 2012, 6:15pm

re: Palo improvements in Kettle 4.3

Thank you for your interest Robert.

Palo and Kettle 4.3 are on the same machine;

Testing of the connection from Kettle to Palo is:

Source code

 
1

"I'm sorry, we can't create a test report for this database.Error connecting to database [PaloDemo] : java.lang.ClassNotFoundException: org.pentaho.di.palo.core.PaloHelperjava.lang.ClassNotFoundException: org.pentaho.di.palo.core.PaloHelper..."


Many thanks for any advise on where to get this working.

Kind regards, DMurray3
Kind regards, DMurray3

tish1

Sage

Posts: 777

Date of registration: Jul 13th 2009

Location: Vienna / Austria

Occupation: Senior Consultant @ Vector SW DV GmbH

  • Send private message

12

Saturday, September 22nd 2012, 7:56am

Hi,

maybe you're working with the wrong PDI steps. I don't know if Pieters adjustment (especially the switch to PaloJLib) are already in the official release.

Regards.
Robert Tischler
Senior Consultant Vector SW DV GmbH

DMurray3

Trainee

  • "DMurray3" is male

Posts: 4

Date of registration: Aug 10th 2009

Location: Quito, Ecuador

Occupation: BI Consultant

  • Send private message

13

Tuesday, September 25th 2012, 1:09am

Pieters van der Merwe´s Improvements to Kettle.

Quoted from "pvanderm"

Thanks for the update.

I'll give it a try and if there are huge benefits from using the new API, I may be useful to use it now. If not, I'll wait for the official release in 3.3.

Quoted from "pvanderm"

Hi Guys

I recently made some big improvements to the Palo plugin in Kettle that will be released with Kettle 4.3.

For those who don't know the history, the plugin was maintained separately to Kettle on SourceForge, but has been merged into Kettle some time ago. The plugin is now maintained by the Kettle community and is included with a standard Kettle installation. You still need to copy the jpalo.jar file to the libext directory since the jpalo GPL licence stops it from being packaged with Kettle.

Improvements include:
1. New Cube Delete Step.
2. New Cube Create Step.
3. Support to specify the consolidation factor when creating dimension elements.
4. Support for the dimension output to return base elements only.
5. Support to recreate a dimension (because a delete all can take forever).
6. Speed improvements on the Cell output by using a caching mechanism and batching updates.
7. Speed improvements on the Dimension output by using a caching mechanism.

A couple of bugs were also fixed.

It has been tested using Palo 3.2, 3.1 and 2.5.

For a full list and details you can go to jira.pentaho.com and search for "palo". You can optionally also filter on "Pentaho Data Integration" to exclude stuff from other projects.

I need some testers to test if the changes I made work and have the desired results. If you are using kettle at the moment, please give it a go. You should appreciate the speed improvements if you are loading a decent sized dataset.

The new Kettle version 4.2 has just been released. No date has been set on 4.3, so you will need to download a CI version from http://ci.pentaho.com/job/Kettle/

If there are any bugs or functions you would like to see in Kettle, please log it on jira.pentaho.com

Kind regards,
Pieter
Dear Pieter,

I am trying to use Pentaho Kettle 4.3 and or 4.2 to create an ETL to upload dimensions, and data to a Palo Cube.

Both updated version of Kettle, alert to include jpalo.jar (which I can´t find anywhere) in Kettle's \libext folder. I understood from other posts, that jpalo.jar was changed to palojlib-1.0.35.jar and from what I gather from your contribution to http://jira.pentaho.com/browse/PDI-6852, I should be using that in Kettle as indicated.

I have placed the palojlib-1.0.35.jar in the \libext of both PDI 4.2 and the recently available version of 4.3, and still cannot get Kettle's Palo Dim Imput or Palo Cell Input steps to work.

Please be advised, that I have tried this on both a PC with W7 64bit (Kettle is in fact working in 64bit mode) and as well as on another PC with W/XP 32bit.

Can you advise where the problem could be and / or what should be done to get Kettle to work/recognize my Palo cubes?

Many thanks for your support and feedback.

Kind regards,
Daniel Murray (DMurray3)
Kind regards, DMurray3

Similar threads

Rate this thread