CSV FILE WRONG DATA

    Hello there,i got a shelly 3EM configured with 3 phases here in greece,i was using Mqqt data of the shelly 3em for a long time when suddenly my home assistant pc crashed without any backup,what does that mean? That means when i fresh installed everything i was getting VIA mqtt the last total data of each phase in (WH).

    Im trying now to merge the old data with the new data,im doing that via downloading the csv file from each phase and format it such as the influx db wants.The problem is that the shelly Csv data Is not correct,im getting more WH from the csv file with 3h of data missing than the actual "LIVE" mqtt feed

    Code
    kWh,domain=sensor,entity_id=3em_l1_total_kwh_mqtt value=2281207.19 1663333980 <- THIS iS THE LAST VALUE OF L1 FROM THE SHELLY CSV WITH 3H DELAY FROM MQTT
                                                total = 2214167.3 <---AND THIS IS THE LAST MQTT MSG I GOT...

    How is it posible to have more kwh consumption before than now...something is really wrong


    On the L3phase im Getting this difference.

    Code
    ,domain=sensor,entity_id=3em_l3_total_kwh_mqtt value=10618697.83 1663345320
kWh,domain=sensor,entity_id=3em_l3_total_kwh_mqtt value=10618767.33 1663345380
kWh,domain=sensor,entity_id=3em_l3_total_kwh_mqtt value=10618837.03 1663345440
kWh,domain=sensor,entity_id=3em_l3_total_kwh_mqtt value=10618906.63 1663345500 <- THIS IS THE LAST VALUE OF L3 FROM THE SHELLY CSV WITH 3H DELAY FROM MQTT
                                                        10925978.8             <----------------- THIS IS THE LAST VALUE OF L3 FROM MQTT JUST MINUTES AGO
                                                        300KWH MORE!!! HOW?

    My point is that something really bad happens when we are downloading the csv files,developers must check this and there is no solution to this,im putting this right here so the devs can check it,i run the same procedure to my friends shelly 3em and he has the same issue,nobody has mention this before (at least as much as i searched nobody had this problem) cause nobody tryed to import csv files into influxDB.

    The conclusion is that the CSV data of EACH phases are wrong,Or the mqtt is wrong but i doubt that.

    P.S i check the format is something is missing and its not,i checked the original CSV file with excelll and i have the same reasult,Its just a wrong algorithm that downloads and build the Wh data on the shelly? i dont know..

    Hi,

    I recently bought a 3EM and face a similar issue. Live figures from 3em look fine but if I down load the history values via .csv file they are much smaller and do not match at all the live values.

    I would like to have some daily graphs of my energy usage but file download delivers wrong values. Trying to get my 3em into my local WLAN it hangs up (but worked fine for my shelly plug S with good live figures and stats in https://home.shelly.cloud/ but 3em stays offline). I tried already all from reboot, reset, .... only fixed IP I did not try yet.

    Any solution available?

    As live data is fine on my EM3 wlan are there any easy and free tools/apps to stream live data to create the graphs? Kalougear you mention you use home assistant with mqtt, would this be something for my use case?

    I have similar issues.

    I downloaded all 3 Phase logs and they look strange (for me...).

    I seems, that the order of the dates in the File of Phase 2 (Figure 2) is wrong.

    I plotted the Time of all 3 Phases (L1 L2 L3) - red the Hours, blue the Minutes.

    Y-Axis shows minutes and hours.

    Figure1 = L1 -> em_data2_l1.csv

    Figure2 = L2 -> em_data2_l2.csv

    Figure3 = L3 -> em_data2_l3.csv

    pasted-from-clipboard.png


    pasted-from-clipboard.png


    pasted-from-clipboard.png

    How did you read that data into Matlab? If you look at the CSV using a text editor there are some things to take care about

    * In general the data is not sorted by timestamp, you have to do that. That looks like being the cause for the slower ramp being at different positions. I'm not sure if Matlab is capable to process the time stamp written into that files.

    * Data is shown in 1 minute intervals for some recent days, looks like that's being the part where your ramps look slower. The remaining data is stored in 10 minute intervals* consumption data is saved as Wh withing the respective time interval. So if the time interval is 10 minutes you will find 10/60 = 1/6 of the consumed power, if the timer interval is 1 minute just 1/60!

    * Maybe you number format and /or delimiter is wrong. This Matlab might import the decimal places as seperate column or ignore them

    * The data recorded might include text like "undervoltage", "overvoltage" or "N/A" instead of numbers in any column, which also might lead to errors parsing the CSV with the standard functions.

    I have done a similar import in GNU Octave (a free "Matlab clone") - not able to deal with all special cases seen in the logfiles so far.

    Best regards