You are not logged in.

import more and more slow

Dear visitor, welcome to Palo Community Forum. If this is your first visit here, please read the Help. It explains in detail how this page works. To use all features of this page, you should consider registering. Please use the registration form, to register here or read more information about the registration process. If you are already registered, please login here.

sebhm

Sage

  • "sebhm" started this thread

Posts: 90

Date of registration: Sep 12th 2008

  • Send private message

1

Monday, September 28th 2009, 5:56pm

import more and more slow

Hi,

first import is fast
but last is slow....
The speed goes down during the successive imports ... why ?
(i'm using SDK PHP)

Is it because of the memory ? the cube PALO ?

Thanks

axi

Sage

Posts: 661

Date of registration: Mar 5th 2009

Location: Germany / Bargteheide

Occupation: BI

  • Send private message

2

Tuesday, September 29th 2009, 9:19am

Maybe because Palo tries to keep the logs in memory and running out of physical RAM it swaps to disk.

You should really use the monitoring tools of your OS to determine the bottleneck.

Restarting Palo might help (stopping writes the logs to disk -> archived, starting starts with clean=empty logs).

sebhm

Sage

  • "sebhm" started this thread

Posts: 90

Date of registration: Sep 12th 2008

  • Send private message

3

Tuesday, September 29th 2009, 10:09am

Thank you.

I have used my monitoring tools, but i just see that memory and processor are fully used when PALO works... but after ..??

For the logs, is it an option for the start of Palo.exe ?
Where can i find a manual with option... ?

Thanks

sebhm

Sage

  • "sebhm" started this thread

Posts: 90

Date of registration: Sep 12th 2008

  • Send private message

4

Tuesday, September 29th 2009, 6:15pm

When i import one file, it may take 10 minutes.
And if i import the same file just after, it takes 10 seconds !

Why ?
because the combinations (the coordonates, i dont know how you call that) are in memory ??

Can't we simule the load in memory before importing the file in order to accelerate the process ?

Thanks

sebhm

Sage

  • "sebhm" started this thread

Posts: 90

Date of registration: Sep 12th 2008

  • Send private message

5

Sunday, October 4th 2009, 2:21pm

you said

Quoted

starting starts with clean=empty logs

is it an option ?

I found this only :

Quoted

usage: palo.exe [-?|--help] [-A|--auto-load] [-a|--admin <address> <port> ...]
[-B|--auto-commit]
[-b|--cache-barrier <number_of_base_elements_in_aggregation>]
[-C|--chdir] [-c|--cache-size <total_cache_size_in_bytes>]
[-D|--add-new-databases] [-d|--data-directory <directory>]
[-e|--clear-cache <number_of_cache_invalidations>]
[-E|--extensions <directory>]
[-g|--clear-cache-cells <number_of_cells>]
[-H|--https <port> ...] [-h|--http <address> <port> ...]
[-i|--init-file <init-file>]
[-K|--key-files <ca> <private> <dh> ...]
[-L|--splash-limit <error> <warning> <info> ...]
[-l|--legacy <address> <port> ...]
[-m|--undo-memory-size <undo_memory_size_in_bytes_per_lock>]
[-n|--load-init-file] [-o|--log <logfile>]
[-p|--password <private-password>] [-R|--user-login]
[-r|--rule-cache-size <total_cache_size_in_bytes>]
[-t|--template-directory <directory>]
[-u|--undo-file-size <undo_file_size_in_bytes_per_lock>]
[-V|--version] [-v|--verbose <level>]
[-w|--worker <worker-executable> <argument1> <argument2> <argumentX> ...]
[-X|--encryption <encryption-type>]
[-x|--workerlogin <worker-login-type>] [-Y|--use-cube-worker]
[-y|--enable-drillthrough]
[-z|--goalseek-timeout <miliseconds>]
[-Z|--goalseek-limit <number_of_cells>]
[-F|--friendly-service-name <service-name>]
[-S|--service-name <service-name>] [-s|--start-service]

<encryption-type> can be 'none', 'optional', or 'required'
<worker-login-type> can be 'information', 'authentication', or
'authorization'. Setting <address> to "" binds the port to any
address. By default palo will change into the data directory before
opening the log file.



I'm looking for any help on Palo Server Behaviour and parameters... ?(

axi

Sage

Posts: 661

Date of registration: Mar 5th 2009

Location: Germany / Bargteheide

Occupation: BI

  • Send private message

6

Monday, October 5th 2009, 12:45pm

The Palo documentation has a chapter 14.1 (Backup) that writes it up nicely:

data is kept in csv-files
logs (changes to your data) are written into log-files

saving the cubes (or shutting down the server) writes the (old) csv to archived-files and the logs into the csv-files and clears the logs

sebhm

Sage

  • "sebhm" started this thread

Posts: 90

Date of registration: Sep 12th 2008

  • Send private message

7

Thursday, October 8th 2009, 9:11am

Thanks,

but it 's always slow...
(an import of 12 data can take more than 1 minute ! )

sebhm

Sage

  • "sebhm" started this thread

Posts: 90

Date of registration: Sep 12th 2008

  • Send private message

8

Tuesday, October 27th 2009, 11:04am

SO...?

Nobody have the same problem ?
how many data do you import in one Palo Base ?
How much Time does it take ?

AWW

Professional

Posts: 19

Date of registration: Dec 23rd 2006

  • Send private message

9

Tuesday, October 27th 2009, 1:40pm

importing

i am importing roundabout 7 Mio Cells with kettle into a cube with 7 Dimensions.
the import takes 600 Cells per second.

sebhm

Sage

  • "sebhm" started this thread

Posts: 90

Date of registration: Sep 12th 2008

  • Send private message

10

Tuesday, October 27th 2009, 2:04pm

Thanks.

I don't use Kettle to import.
I use functions of SDK (in PHP), I'm going to look how Kettle works.

sebhm

Sage

  • "sebhm" started this thread

Posts: 90

Date of registration: Sep 12th 2008

  • Send private message

11

Tuesday, October 27th 2009, 2:25pm

Maybe one link to Doc, Tutorial or anything about Kettle and Palo ?
Rate this thread