Palo Community Forum

Hi all,

we want to go live next time with a palo system, which stores a large amount of data (what in-memory olap is said to be predestinated ;-). Some dimensions have round about 10K elements and are using many palo.data rules towards to the final result cube used for reporting.

We now test on a 4 Xeon Core/32 GB Ram machine (Amazon AWS), and when I stress the server with an request crossing all of these large dimensions it comes to HTTP 500 error or at least to a huge runtime no real user would wait for (tens of minutes...). But nevertheless the server is bored: CPU workload never above 30%, RAM usage never above 25% of available amount... so the hardware is more than sufficient.

In the palo.ini there are some switches about cache, but I don't find any description (even not in the setup guide), how to configure my server machine in a scenario like above...

Does somebody have experiences with this stuff? Are there some things to do to make Palo works faster by consuming more hardware resources so far available?

Our licence is a PE 3.3SR2 on Windows 2008 SRV STD (paid stuff!), but as our Administrator indicated, this don't seem to cover some support or advanced documentation for it

Any hint is very much appreciated

Best regards,

Stefan

Hi all, thanks very much for your replies :-)

Yes, all CPUs are utilized equally, but the GPU-approach would be very expensive (I read before,just the GPU-subsystem could cost up to 50T€...). The rule-cache size made a small improvement when turned it higher, but we have no idea what could be the best value to use... I expected, that all hardware resources would be used until 100% utilized and therefor be the limiting factor, but as we learn now, it is not.

To avoid a huge amount of data and rules was my aim also, but the customer insisted that he really need the multi-step-approach to calculate the reported values at this granularity at all -> maybe I need to propose really the GPU-solution, hopefully they remember their priorities then ;-) We precalculate as possible in ETL, but then 3 (and up to 6 next stage) cubes load the data and calculate then the next interim results and these then are combined in the master cube which is the base for the reports. And besides we have 2 lookup cubes for categorization purposes, one of them has 10K elements on the largest dim. At all, this project would be moreover a case for TM1, but they prefer open source and wanted to give PALO a try...

Our frontend is Jedox Web (spreadsheets), but not dedicated Excel via AddIn, this I only use for development purposes (Adv. Rule Editor mostly).

The retrieved data are numbers, not strings. But many palo.data-rules (accessing to data/interim results from upstream stage cubes) use string formulas (I need to calculate a %-value, round it to 1 digit after comma and match it then to a lookup-cube, which gives back a category number. And in two cases a >= and <= check results in a result stored as a string resp. an attribute is pulled into result cube besides a dimension value, because later I want it to mark in a traffic light style.
So 3 different calculations are done and matched to a category (Interim Calculation cubes A-C with matching to Category Lookup Cube). Then these 3 interim results are referenced via palo.data-rules into the Master Cube, which itself makes a lookup to the Value Weight Cube and calculates on the 3 weighed interim values a master value. The master cube contains also one more measure which is gained by dump summing up, not by cube references.

Amount of retrieved data:

Cube/type of value Value from Cube properties (Modeller)
Size A-C (structure identically) 44460976290228000
Interim Calculation Cube A filled cells 41230
Interim Calculation Cube B filled cells 20457
Interim Calculation Cube C filled cells 60604
Master Cube Size 277881101813925000
Master Cube filled cells 92837
Category Lookup Cube Size 1887510
Category Lookup Cube filled cells 1851
Value Weight Cube Size 170
Value Weight Cube filled cells 38

There is very much sparsity, so I tend to try markers now, but as I read in the advanced Excel manual, this is useful only when I have a calculation and one component of the formula could be 0.
But how fits this to the palo.data-rule?
There is also shortly described @palo.marker, but could i simply pin this on my several palo.data-rules and use the compete set of dimension names like in the example in the manual?
Or do I need to marker only subsets of the source cubes? Does it work exactly along the same concept as TM1's Feeders do?
For the productive phase planned in the near future, we plan to load more data, approx. 3 times compared to now, which will substitute the sparsity space to filled space, because the dimensional structures are complete right now.

Much stuff, I apologize for that, but I hope thus I can forward the appropriate information for a helpful hint , which would be VERY appreciated

Best regards,
Stefan

Ok, then I should try the markers, because I work for several measures with a =B:palo.data... x (local value) to fill them.

My major gap of understanding is, what to marker. I think, its a reversal of retrieving data in case of much sparsity, so i focus the engine to work only on the filled cells instead of all cells (when almost filled and seldom empty). I do this by pointing on the area where to find filled cells in the wide lonesome and empty space of the cube. But the example of the Excel advanced manual available from Jedox ...



...seems to marker the complete source cube, not only a slice or so. And the target here is the measure Umsatz (revenue for non-german speakers), and only for the year 2005. But how to marker the target ['2006', 'Umsatz'], do I need to create for each year a separate rule with marker? If I have 10 years, it could be a little work to duplicate this, nevertheless I would survive that, but somewhen data for the year 2007 will be loaded from DWH. This would not be markered then, do I need to expand my rule collection manually again and again? I designed my cubes actually in that way, that they fit to the business requirements independent of something hardcoded, fully metadata-driven... is this a lost option when needing markers?

If I changed the target in the Jedox example to ['Umsatz'], will I encounter a data explosion over the years? Or could I expect the rule to work like without the marker and source=2006 leads only to a value in ['2006', 'Umsatz'] and same for each other year...?

Did somebody ever write or found somewhere a kind of tutorial or whitepaper or so about that marker stuff which explains it more in a step-by-step fashion and not just by a (simple) example? Or is the Jedox manual really the only written knowledge about that topic until now... ?

@sql8: persisting of the data we discussed also, and it would be possible due to a loading frequency of 1x / week. The whole 10K amount of elements at some dimensions occurs due to flexibility needs (whishes...) of the users, I could not convince with my proposal to reduce this to a more rough structure... so we will have to work this way, I guess.

Many thanks for your replies,

Stefan