Probe Logging Performance Issues

[tweedge]

Hi. When working with my WiFi Pineapple NANO, the longer PineAP is on and logging probes the less data seems to be collected. I presume this is because of two things:

1. I have the SQLite database stored on a MicroSD card instead of on the internal flash, which presumably incurs an IOPS penalty
2. Every time a duplicate probe is collected, all prior duplicates of that probe must have their "dups" (duplicates?) field updated

This becomes extremely problematic when a large number of disparate probes have been logged, and there appears to be a severe performance penalty incurred beyond 50,000 logged probes, which slows down future data collection. Would it be possible either now via manual modification or in the future via firmware update to simply remove the dups counters? I can easily count the duplicates after the fact, and don't see this information presented in the UI.

Please see the attached photo of a Kibana graph.

Let me know if any of you have experienced the same, or could work around it. Thank you!

[tweedge]

@Foxtrot any guidance on getting this reproduced/triaged by Hak5? The bugtracker seems to be down, and I am still seeing this issue. Clearing out the sqlite DB works in a pinch, but I can't make a habit of this 😞 

[Foxtrot]

Hey tweedge,

The duplicates counter is used internally, and is also displayed in the Logging module in the UI. I will look into the issue you're experiencing when I can.

Thanks

[tweedge]

Huh - didn't notice it in the Logging module. Thank you for the guidance! I'll keep my eyes open for updates.