Coding: Fleeting Thoughts

A place to discuss the implementation and style of computer programs.

Moderators: phlip, Moderators General, Prelates

User avatar
Flumble
Yes Man
Posts: 2263
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Fri May 03, 2019 3:39 pm UTC

Make some kind of ziplist iterator that provides immutable references to all previous and next items while providing a mutable reference to the current item. :D

Would it make sense to map over the collection producing a collection of Option<Mesh> objects (Some if a mesh is generated) and then zip the two collections? Or can the (re)generation of one mesh depend on the (re)generation of another mesh during the same iteration?

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Fri May 03, 2019 10:00 pm UTC

Flumble wrote:Would it make sense to map over the collection producing a collection of Option<Mesh> objects (Some if a mesh is generated) and then zip the two collections? Or can the (re)generation of one mesh depend on the (re)generation of another mesh during the same iteration?

Yeah, I can push all the changes to a separate collection, then merge the changes back into the original collection as a separate pass. That's what I implemented for now. It's cheap compared to the mesh generation, but it's still pointless overhead just to appease the compiler.

I was promised readable idiomatic code with zero-costs abstractions, and now I need to sacrifice both readability and performance just to make it compile. Not sure how I feel about that.

Next step is to implement the actual Octree in Rust. Let's see how pretty the solution for a recursive data structure ends up. I'm anticipating more pain because each object must be both in a HashMap (based on id) and in the Octree (based on location), and although I can guarantee that to be safe, I'm afraid the compiler won't believe me unless I accept the overhead of ref-counting smart pointers.

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Sat May 04, 2019 9:00 pm UTC

Update: I ended up solving the mutability problem by using std::Cell. Cells cheat the whole lifetime issue by disallowing getting a reference to the things inside. You can just copy values in, or copy the current value out. With those semantics, it's safe for them to allow changing the value inside without having a mutable reference to them! Now I can do the whole iteration and regeneration on immutable objects, but still clear the dirty-bit inside a Cell<bool>. Won't solve all of my problems, but for this specific problem the borrow checker is happy.

Octree went well. I needed a bit more boilerplate to specify the type of child pointers (either nullptr, pointing to another node, or pointing to a leaf with objects), but thanks to enums that was a lot less code than other statically typed languages would have required.
For now the octree leaves just store the object ids instead of references to the objects, neatly avoiding all the borrowing issues. That's another HashMap lookup per visible object per frame, but those lookups are cheaper than drawing the object, so I guess it's fine. Still irks me, but it's fine.

The previous javascript implementation used a Heap, added all the visible objects by distance, then continued to draw them front-to-back. In rust, I ended up implementing this as a lazy iterator, allowing me to consume items before the Heap is completely filled. Not sure if that's actually faster, but it was a fun challenge.
The Iterator required a ton of boilerplate for all the types, their trait implementations and everything (the BinaryHeap requires 4 traits on its entries just to specify sort order!), but the iterator was really straightforward. Pop the closest item from the queue; if it's an Octree node add all the children to the queue (frustum culled), if it's an Object return it. Match expressions really shine here.

It's also fast. Counting the time between starting the game and watching the last object pop in, the wasm implementation appears to be ~20-50% faster. Success.

Now for the difficult part: not losing interest in the project after finishing the proof-of-concept. :roll:

User avatar
Flumble
Yes Man
Posts: 2263
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Tue May 07, 2019 12:12 pm UTC

Can I get a smartphone (android/ios) to keep javascript and location services active while the screen is off? I'd like to track my (and friends') running and send it to a server in realtime, preferably from a webpage (stores are expensive and sideloading is hard) and without having the phone in drainage mode.

Most of what I found so far are stackoverflow questions of 5+ years ago.


Tub wrote:Now for the difficult part: not losing interest in the project after finishing the proof-of-concept. :roll:

The goal is the proof-of-concept, right? :mrgreen:

User avatar
Xanthir
My HERO!!!
Posts: 5425
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Coding: Fleeting Thoughts

Postby Xanthir » Tue May 07, 2019 3:09 pm UTC

Flumble wrote:
Can I get a smartphone (android/ios) to keep javascript and location services active while the screen is off? I'd like to track my (and friends') running and send it to a server in realtime, preferably from a webpage (stores are expensive and sideloading is hard) and without having the phone in drainage mode.

Most of what I found so far are stackoverflow questions of 5+ years ago.

No, you can't. It's a sensitive operation that we're not willing to allow a page to push into a background Service Worker and invisibly track you with; the page needs to be open and active to receive location data.

Native apps are just security/privacy nightmares.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

User avatar
Flumble
Yes Man
Posts: 2263
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Tue May 07, 2019 10:28 pm UTC

Xanthir wrote:No, you can't. It's a sensitive operation that we're not willing to allow a page to push into a background Service Worker and invisibly track you with; the page needs to be open and active to receive location data.

You standards people always ruin everything!!1! First you can't just retrieve a precise location without the user's permission, now you can't even track people whenever you like! :roll:

I guess I'll go with "keep your phone unlocked constantly" for now. The reason I want to send the position in real time in the first place, is so you can see each other on the map in real time, for which you'll need the screen turned on anyway. (Telling the relative distance would be neat too, but with that much overengineering I may as well put in the effort of turning it into a native program that can access the location in the background.)

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5101
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Wed May 08, 2019 3:42 am UTC

I can't decide if this is beautiful or horrifying:

Code: Select all

private static string GetFullName(ClassDeclarationSyntax @class)
{
    string fullName = @class.Identifier.ValueText;
    SyntaxNode node = @class;
    while(true)
    {
        switch (node.Parent)
        {
            case NamespaceDeclarationSyntax @namespace:
                fullName = $"{@namespace.Name}.{fullName}";
                break;
            case ClassDeclarationSyntax outerClass:
                fullName = $"{outerClass.Identifier.ValueText}.{fullName}";
                break;
            default:
                return fullName;
        }
        node = node.Parent;
    }
}

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Thu May 09, 2019 3:29 pm UTC

TIL that google will happily store and monetize all of your location data, but considers it a privacy problem when someone else does. :roll:

Xeio wrote:I can't decide if this is beautiful or horrifying:

It can be used for good or evil. I wonder, which is it?

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Sat May 18, 2019 10:00 pm UTC

Tub wrote:It's also fast. Counting the time between starting the game and watching the last object pop in, the wasm implementation appears to be ~20-50% faster. Success.

I wasn't happy with this. A bit of benchmarking later, the mesh generation code has been improved by another +120% on top of the previous improvements.

I wouldn't know how to do these kinds of optimizations in a high level language (like JS) when targeting an unknown and changing environment (like a browser).

So my thoughts of the day are:
  • If you need speed, and you're willing to invest the time, wasm works.
  • Hash maps are at the core of most dynamic languages, used to implement objects. Lookups are O(1) and presumed to be really fast. However, when you're doing ~150.000 of them per frame, it helps to bring that number down, and it also helps to use a faster (but less secure) hashing function than the default one.

User avatar
Flumble
Yes Man
Posts: 2263
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Mon May 27, 2019 4:48 pm UTC

Code: Select all

const /*or let or var*/ variable = someFunctionOf(variable)

The javascript complaineth. I am sad. Though recursive binding is allowed for the "old" function syntax:

Code: Select all

function variable() {
  return someFunctionOf(variable)
}

Sure, let v=v+5 would make a mess, but in my case someFunctionOf behaves perfectly fine (e.g. (_)=>0) ...if javascript would create a(n undefined) reference for variable first and afterwards replace it with the result of the expression. Changing a reference externally is not weird un-javascript behaviour at all. :roll:

So when can we expect a letrec keyword in ecmascript?

User avatar
hotaru
Posts: 1045
Joined: Fri Apr 13, 2007 6:54 pm UTC

Re: Coding: Fleeting Thoughts

Postby hotaru » Tue May 28, 2019 12:31 am UTC

Flumble wrote:Though recursive binding is allowed for the "old" function syntax:

Code: Select all

function variable() {
  return someFunctionOf(variable)
}


the equivalent of that code with const does work, though:

Code: Select all

const variable = () => someFunctionOf(variable);

Code: Select all

factorial product enumFromTo 1
isPrime n 
factorial (1) `mod== 1

User avatar
phlip
Restorer of Worlds
Posts: 7573
Joined: Sat Sep 23, 2006 3:56 am UTC
Location: Australia
Contact:

Re: Coding: Fleeting Thoughts

Postby phlip » Tue May 28, 2019 6:04 am UTC

it can be interesting to see which dynamic languages let you do this, and which ones don't...

Code: Select all

phlip@boris:~$ python
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = str(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined

phlip@boris:~$ irb
irb(main):001:0> a = a.to_s
=> ""

phlip@charlemagne:~$ js
> var a = '' + a;
> a
'undefined'
> let b = '' + b;
ReferenceError: b is not defined
> const c = '' + c;
ReferenceError: c is not defined

Code: Select all

enum ಠ_ಠ {°□°╰=1, °Д°╰, ಠ益ಠ╰};
void ┻━┻︵​╰(ಠ_ಠ ⚠) {exit((int)⚠);}
[he/him/his]

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue May 28, 2019 9:07 am UTC

Flumble wrote:

Code: Select all

const /*or let or var*/ variable = someFunctionOf(variable)

This works does not explode with var. Anything declared with var has a scope of the whole function, being initialized to undefined before its first assignment. It doesn't work with const or let because of their restricted scope and temporal dead zones.

Flumble wrote:

Code: Select all

function variable() {
  return someFunctionOf(variable)
}

function declarations also have function-level scope; you can reference them before, after, and even during declaration.

However, temporal dead zones still apply for captured variables:

Code: Select all

  let foo2 = foo; // referencing works fine
  foo(); // calling it would access x early; this throws.
  let x = 42;
  function foo() {
    return x;
  }
  foo(); // now it works


Flumble wrote:So when can we expect a letrec keyword in ecmascript?

Step #1 would be to show some kind of problem that's easier to solve with the keyword than without. Considering that it works with var, I doubt you'll find one.

Otherwise, Babel is a pretty sweet project. Adding a plugin to implement your favourite code transformation isn't hard.

phlip wrote:it can be interesting to see which dynamic languages let you do this, and which ones don't...

Code: Select all

 ~> php -r '$a = gettype($a); echo $a;'
PHP Notice:  Undefined variable: a in Command line code on line 1
NULL

Depending on your settings (and PHP version), it may output a notice, but otherwise continues as if the variable was NULL.
Last edited by Tub on Tue May 28, 2019 9:14 am UTC, edited 1 time in total.

User avatar
chridd
Has a vermicelli title
Posts: 846
Joined: Tue Aug 19, 2008 10:07 am UTC
Location: ...Earth, I guess?
Contact:

Re: Coding: Fleeting Thoughts

Postby chridd » Tue May 28, 2019 9:11 am UTC

perl:

Code: Select all

$ perl -e 'my $x = $x.""; print $x;'
(no output)
...but...

Code: Select all

$ perl -e 'use strict; my $x = $x.""; print $x'
Global symbol "$x" requires explicit package name at -e line 1.
Execution of -e aborted due to compilation errors.
...so at least Perl treats $x in that statement as referring to a global variable in an outer scope.

Also, on my browser, your JavaScript examples give "ReferenceError: can't access lexical declaration `b' before initialization". And

Code: Select all

const d = function() {return d;};
works.

Also, function statements execute before any other statements, so you can call a function before it's defined.
~ chri d. d. /tʃɹɪ.di.di/ (Phonotactics, schmphonotactics) · she · Forum game scores
mittfh wrote:I wish this post was very quotable...

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Wed May 29, 2019 10:45 am UTC

I just realized that javascript's Function class has a unique property. All constructors are functions, so they inherit from Function. This includes the Function constructor:

>> Function instanceof Function
true

A small challenge is thus: create a new variable A (with A !== Function) such that
>> A instanceof A === true

This is also possible in PHP, but requires a different approach. I don't know about python or perl. Not sure if Java or C# and their Reflection APIs allow something to be both a variable and a type.

User avatar
Flumble
Yes Man
Posts: 2263
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Wed May 29, 2019 4:35 pm UTC

hotaru wrote:the equivalent of that code with const does work, though:

Oof, that was a blunder. :oops: Yes, very true. (And in both these cases getting something out of variable would require evaluating the function again and again, which would defeat my usecase) Perhaps I've seen too much haskell in the past weeks, forgetting what eager evaluation looks like.

phlip wrote:it can be interesting to see which [...] languages let you do this, and which ones don't...

I'd recommend using an object/list/dictionary to give the language a bit of leeway. Who knows, maybe there's a language that implicitly stores a reference:

Code: Select all

$ python3 -c "a = {'key': a}"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'a' is not defined

$ echo 'a={"key" => a}' | irb2.5
Switch to inspect mode.
a={"key" => a}
{"key"=>nil}

$ js -p 'var a = {"key": a}; a'
{ key: undefined }

$ perl5.26.1 -e 'use Data::Dumper; my %a=("key"=>%a); print Dumper \%a'
$VAR1 = {
          'key' => undef
        };

$ javac Main.java
Main.java:4: error: self-reference in initializer
   static final Map<String,Object> map = Map.of("key",(Object)map);
                                                              ^

//this is cheating, really
$ printf '#include<iostream>\n#include<map>\n typedef const std::map<std::string,const void*> Map; int main() { Map map = {{"key", &map}}; std::cout << ((Map*)map.at("key"))->begin()->first << std::endl; return 0; }' | g++ -Wall -Wextra -x c++ -
//no errors

$ php7 -r '$a = array("key"=>$a); echo gettype($a["key"])."\n";'
NULL
//could also cheat with '$a = array("key"=>a); echo gettype(${$a["key"]});'

$ ghc -e 'let a = "value":a in print a'
["value","value","value","value","value","value","value","value","value","value","value","valu^C

(By the way, java doesn't even allow a = () -> a.)
Lisp and friends do have recursive bindings (it's what letrec was born to do), but I guess no language on the imperative side can do it.

Tub wrote:A small challenge is thus: create a new variable A (with A !== Function) such that
>> A instanceof A === true

Is it going to involve defineProperty hacks?
Spoiler:
Oh, I missed one trivial case A=Object

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Wed May 29, 2019 7:29 pm UTC

Oh. I missed that rather trivial solution, too. It's true, this will also work for all of Function's parents, which is just the one you mentioned.

But no, it can be solved without reusing an existing constructor. I don't think defineProperty will help much.

DavidSh
Posts: 217
Joined: Thu Feb 25, 2016 6:09 pm UTC

Re: Coding: Fleeting Thoughts

Postby DavidSh » Wed May 29, 2019 8:41 pm UTC

In python, both isinstance(type,type) and isinstance(object,object) evaluate to True. You can also set isinstance() for an object to do pretty much anything you want.

Code: Select all

class x(object):
  def __instancecheck__(self,y):
      return True
     
xx = x()

isinstance(xx,xx)
isinstance(3,xx)

Anything is an instance of xx generated in this way.

User avatar
phlip
Restorer of Worlds
Posts: 7573
Joined: Sat Sep 23, 2006 3:56 am UTC
Location: Australia
Contact:

Re: Coding: Fleeting Thoughts

Postby phlip » Thu May 30, 2019 12:33 am UTC

Python2 old-style classes are also just total jank and you can do whatever the hell you want:

Code: Select all

>>> class A:
...   pass
...
>>> A.__class__ = A
>>> isinstance(A, A)
True

Code: Select all

enum ಠ_ಠ {°□°╰=1, °Д°╰, ಠ益ಠ╰};
void ┻━┻︵​╰(ಠ_ಠ ⚠) {exit((int)⚠);}
[he/him/his]

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue Jun 04, 2019 9:50 am UTC

Going to resolve this real quick.

Quoting mdn on instanceof:
The instanceof operator tests whether the prototype property of a constructor appears anywhere in the prototype chain of an object.

For A instanceof A, you need A to
* be a constructor (a function) and an instance
* as a constructor, have a prototype (aka parent class), which is A.__proto__ (nonstandard) or Object.getPrototypeOf(A) (modern)
* as an instance, have that prototype inside of its prototype chain (A.prototype, or further up the chain)

Doing those requirements verbatim, you can do this:

Code: Select all

function A() {}
A.prototype = Object.getPrototypeOf(A);

But then A is just a plain function, not a custom class. If you want your own prototype in there, you'll find that you cannot simultaneously define A as function A() and as A = Object.create(..). But there's Object.setPrototypeOf:

Code: Select all

function A() { };

let proto = { };
Object.setPrototypeOf(A, proto);

A.prototype = Object.getPrototypeOf(A);


In PHP, the trick is a bit different. The operator is defined as [instance] instanceof [class], usually $A instanceof Foo.
[instance] can be a variable $A or a constant A.
[class] can be a class name, or a string variable holding a class name.

Code: Select all

A instanceof A

This might work if A is both a constant and a class name (not sure how PHP resolves the naming conflicts), but constants cannot hold objects, just primitive types. Primitive types are never an instance of any class.

Code: Select all

$A instanceof $A

This works if $A is an object of our class, and a string holding the name of our class. Remembering that strings are primitive types, not objects, can this be done? Thanks to type coercion, it can.

Code: Select all

class Foo {
   public function __toString() {
      return 'Foo';
   }
};

$A = new Foo();

echo ($A instanceof $A) ? 'yes' : 'no';

User avatar
Link
Posts: 1419
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Tue Jul 09, 2019 11:14 am UTC

C++ question, out of interest rather than necessity. Suppose I have something like this:

Code: Select all

#include <functional>

template<bool B = true> class A
{
    typedef A<B> this_type;

    void foo(void)
    {
        auto cb = std::bind(&this_type::bar, this, std::placeholders::_1);
        some_method_needing_a_callback(cb);
     }

    int bar(int x)
    {
        /* ... */
    }
};

I.E. a class template that binds one of its own member functions to use as a callback. Here I've used the typedef to maintain flexibility, so I can change the template parameters at a later time without having to dig through the class to replace the bindings. However, this feels like a bit of a kludge to me, especially because you need to write out the entire list of template parameters in the typedef. Can anyone think of a more elegant way to do this?

User avatar
Flumble
Yes Man
Posts: 2263
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Tue Jul 09, 2019 12:58 pm UTC

The only way I know where you don't need the typedef is using a lambda:

Code: Select all

auto cb = [this](auto x){ return bar(x); };

But it's not elegant.


Unrelated question: is there a standardized method for absorbing old database records into an aggregate?
For example, I have a table (timestamp, user_id, post_id) for new posts. However, after a month I only really care about how many posts a user has, not which posts in particular. So I want to forget the individual records (with timestamps older than a month) and aggregate them in another table (user_id, post_count).*
Are there frameworks that can do this for you (specify the aggregate you want and it generates and updates the table for you, deletes old records and whatnot)? Does this method have a searchable name?

*Don't know if it's better to increase the post count immediately when a post is added or only when it's due for removal, but that should happen transparently anyway. (Doing the former is easy because you don't have to add post_count and the single posts to get the total post count. Doing the latter is easy when you want to undo a post. Well, often the aggregation function is commutative and invertible (removing a post is simply decrementing the post count), so undoing a post is easy either way.)

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue Jul 09, 2019 1:28 pm UTC

Huh. C++ really doesn't have a Self keyword. I've been rust'ing too much.

I'm not aware of any standard compliant way to get that typedef in a static context, but you don't need it:

Code: Select all

auto cb = std::bind(&std::remove_pointer<decltype(this)>::type::bar, this, std::placeholders::_1);

You can also do the typedef for readability, but you must declare it within a non-static function:

Code: Select all

using this_type = typename std::remove_pointer<decltype(this)>::type;
auto cb = std::bind(&this_type::bar, this, std::placeholders::_1);

or just use the lambda if some_method_needing_a_callback() can accept non-std::function types.

Flumble: I believe this is commonly called "Shooting yourself in the foot by de-normalizing your database", but if performance requires it, the queries are simple enough that it shouldn't need a name nor a framework.
All forum software I know increments the post count when posting (and decrements when deleting a post), because that's more performant than doing an index- or table scan each time you need a post count, and it also allows purging of old posts without having to update the postcounters.

For simple counters, sums, mins or max, this is all easy. You can implement averages with a sum and a counter. Do you have aggregations that are more complicated than that?

User avatar
Yakk
Poster with most posts but no title.
Posts: 11129
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Coding: Fleeting Thoughts

Postby Yakk » Tue Jul 09, 2019 2:10 pm UTC

Don't use std::bind as a general rule. It was added to C++ at the same time as lambdas (C++11); in C++11 it could do a few things lambdas cannot.

As of C++14, there is nothing it can do that lambdas cannot.

While you might find std::bind readable, that is because you don't understand that it does insane things if passed seemingly innocuous parameters. It is a bad, bad idea to use in generic code because of what happens when someone passes, say, the return value of std::bind to it, or a bunch of other stuff.

Use lambdas.

Code: Select all

        some_method_needing_a_callback([this](int x){ return bar(x);});

or, if the callback doesn't persist, just use `[&]`:

Code: Select all

        some_method_needing_a_callback([&](int x){ return bar(x); });
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

User avatar
Flumble
Yes Man
Posts: 2263
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Tue Jul 09, 2019 2:57 pm UTC

Tub wrote:Flumble: I believe this is commonly called "Shooting yourself in the foot by de-normalizing your database", but if performance requires it

In my case it's not so much about performance as it is "store only things that are absolutely necessary". I still need a grace period for editing/removing records, but after that only the aggregate information needs to remain.
If the records were blocks on a blockchain (with the assumption that you want to forget the blocks after a grace period) and the aggregate is the head of the chain, removing/changing any block would be impossible if the aggregate contains the true head of the chain rather than the head-of-old-blocks (since the chaining function is not reversible). Now, in my case it's simple and reversible like the post counter, but a proper framework would account for this.

But I'll be damned if there isn't a framework for this in the face of GDPR and other privacy legislation. There are plenty frameworks for "data anonymization (citation needed)", so "aggregate from records with grace period" seems general enough to warrant one too, but I can't find one so far.

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Tue Jul 09, 2019 10:32 pm UTC

Thanks to marketing, I've seen dozens of incompatible definitions of "blockchain", most of them far removed from the distributed algorithm behind bitcoin. But you're the first to use the term blockchain for ephemeral data. In other words, I still have no clue what problem you're actually trying to solve.

Though you might look for something like a data warehouse? That's usually the term for databases where you dump a lot of aggregates for further analysis.

User avatar
Thesh
Made to Fuck Dinosaurs
Posts: 6598
Joined: Tue Jan 12, 2010 1:55 am UTC
Location: Colorado

Re: Coding: Fleeting Thoughts

Postby Thesh » Tue Jul 09, 2019 10:56 pm UTC

This is something easily handled by a database trigger on insert/delete.
Summum ius, summa iniuria.

User avatar
Link
Posts: 1419
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Wed Jul 10, 2019 8:36 am UTC

Yakk wrote:
Don't use std::bind as a general rule. It was added to C++ at the same time as lambdas (C++11); in C++11 it could do a few things lambdas cannot.

As of C++14, there is nothing it can do that lambdas cannot.

While you might find std::bind readable, that is because you don't understand that it does insane things if passed seemingly innocuous parameters. It is a bad, bad idea to use in generic code because of what happens when someone passes, say, the return value of std::bind to it, or a bunch of other stuff.
Huh, I didn't know that. Alright, I'll just switch to lambdas and ditch the whole this_type m*stard then. Thanks!

User avatar
Xenomortis
Not actually a special flower.
Posts: 1455
Joined: Thu Oct 11, 2012 8:47 am UTC

Re: Coding: Fleeting Thoughts

Postby Xenomortis » Thu Jul 11, 2019 9:33 am UTC

If at first you don't succeed, try, try again.

Code: Select all

def try_hard(func):
    def thunk(*args, **kwargs):
        while True:
            try:
                return func(*args, **kwargs)
            except:
                pass
    return thunk

Code: Select all

class Weakling:
    @try_hard
    def whiny_method(self):
        # some bullshit that throws exceptions randomly
        ...


Worked well enough to get something through the night, but I can't say it left a great taste in my mouth.
Image

User avatar
Xanthir
My HERO!!!
Posts: 5425
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Coding: Fleeting Thoughts

Postby Xanthir » Mon Jul 15, 2019 10:13 pm UTC

I would put a max_retries parameter on that, and scope which kinds of errors you want to ignore, at which point you have a fairly reasonable retry() method.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

User avatar
Xenomortis
Not actually a special flower.
Posts: 1455
Joined: Thu Oct 11, 2012 8:47 am UTC

Re: Coding: Fleeting Thoughts

Postby Xenomortis » Mon Jul 15, 2019 11:12 pm UTC

Yeah, it was actually a "for n in range(...)" loop, and in another part of the code base where I've actually used that pattern deliberately, the exception type is constrained. It was left unconstrained here because I wasn't 100% sure what I would actually get (and the retries where basically harmless, if time consuming).
Image

User avatar
Flumble
Yes Man
Posts: 2263
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Wed Aug 07, 2019 7:53 am UTC

Are there any security implications of (symmetrically) encrypting data with a key derived from that data? (e.g. aes-encrypt(data, sha-256(data)), with a given IV where applicable)
Apart from the expected reduced security for data shorter than the key length and replay attacks, that is.

User avatar
ucim
Posts: 6888
Joined: Fri Sep 28, 2012 3:23 pm UTC
Location: The One True Thread

Re: Coding: Fleeting Thoughts

Postby ucim » Wed Aug 07, 2019 2:09 pm UTC

IANASE, but think back to the enigma machine. The key to breaking the encryption was that a letter could not represent itself. Every plaintext letter had to have a different letter for its cyphertext. This allowed codebreakers to rule out certain possibilities, reducing the search space. Because many of the messages contained expected plaintext ("weather", for example), this ruling-out was very powerful.

It would seem to me that similarly, if the key depended on the data, the solution space for the key would be smaller. Certain keys (or even huge swaths of possible keys) might be disallowed, depending on the actual function you used. The way that happened might be subtle, but it would still be there.

For a trivial example, suppose a password could only contain characters that existed in the data, and was randomly generated from their frequency in that data. The password for a stock price table would be simple to find. Your main security here is that the enemy does not know that this is what you did, and that's poor security.

Don't roll your own crypto. But... then you have to trust those who are rolling the crypto you're using.

I have a different question. Is there an adverse security implication to adding your own crypto to a message before it is encrypted with an existing, "known good" crypto? Could one "undo" the other, in part, like ROT 13 would undo itself?

Jose
Order of the Sillies, Honoris Causam - bestowed by charlie_grumbles on NP 859 * OTTscar winner: Wordsmith - bestowed by yappobiscuts and the OTT on NP 1832 * Ecclesiastical Calendar of the Order of the Holy Contradiction * Heartfelt thanks from addams and from me - you really made a difference.

Tub
Posts: 475
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Wed Aug 07, 2019 7:48 pm UTC

Also NASE, I just occasionally use that stuff for mission-critical systems :roll:

The way you derive the key from the data cannot be arbitrary, but using a cryptographic hash as you suggested seems sound. The avalanche effect guarantees properties about the randomness of the output of a cryptographic hash. A CSPRNG must satisfy the next-bit-test. Assuming the input has enough entropy, that looks like it'll work out. I'll leave the formal proof to you, but I wouldn't worry about the result not being random enough.

But now you need to worry about the inputs having sufficient entropy (which is not the same as enough bits), about novel partial known plaintext attacks, about timing attacks due to non-constant CSPRNG runtime, and about a slew of other problems that neither you nor me nor actual security experts can come up with right now.

Don't roll your own crypto, and don't implement your crypto audits with a random forum poll.

ucim wrote:I have a different question. Is there an adverse security implication to adding your own crypto to a message before it is encrypted with an existing, "known good" crypto? Could one "undo" the other, in part, like ROT 13 would undo itself?

An encrypted output should appear random, so it's possible (but just as unlikely as any other output) for it to equal the plaintext. This is quite likely on a scheme with just 26 possible keys, but not an issue with practical key sizes.

So chaining is safe, because you don't even need to guarantee the property you're worried about.

User avatar
Link
Posts: 1419
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Sun Aug 11, 2019 5:06 pm UTC

And so, on this dreary day, I find myself gazing deeply into the ink-black abyss that is Unicode. Already, as I stand on the threshold of the eldritch maelstrom, I feel the light of sanity fading from my mind, and yet, I know there is but one way forward: to leap, in defiance of all primal instinct to turn and run, forward into its vile maw, that I may emerge victorious and grasp the forbidden secrets of text input.

User avatar
Flumble
Yes Man
Posts: 2263
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Sun Aug 11, 2019 5:56 pm UTC

You should follow your instincts on this one. There's nothing but sorrow and regret down there, so forget about Han unification, rendering mixed RTL/LTR text and canonical representations. Instead, enjoy our pirate flag emoji and ͔Z̛̙̦̭͎̦̺A̧̟̝̫L̙͚̦̺̟͕͔G͇̫̲̞̩O͇̹̭̹̗.

User avatar
Xenomortis
Not actually a special flower.
Posts: 1455
Joined: Thu Oct 11, 2012 8:47 am UTC

Re: Coding: Fleeting Thoughts

Postby Xenomortis » Sun Aug 11, 2019 6:32 pm UTC

Link wrote:And so, on this dreary day, I find myself gazing deeply into the ink-black abyss that is Unicode. Already, as I stand on the threshold of the eldritch maelstrom, I feel the light of sanity fading from my mind, and yet, I know there is but one way forward: to leap, in defiance of all primal instinct to turn and run, forward into its vile maw, that I may emerge victorious and grasp the forbidden secrets of text input.

100:1 against you learning the forbidden knowledge.
10:1 on you dying a horrible death.
Image

User avatar
Link
Posts: 1419
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Sun Aug 11, 2019 6:52 pm UTC

:lol:

Looking further into it, it does seem to be a world of pain. Looks like the safest way forward is to just make some sanity-preserving demands about any user-given text, and then just pop it into ICU and trust that whatever it comes up with is reasonable. (And if I'm going to add ICU to the list of dependencies, I might as well bite the bullet and add Boost too, so I can use Boost.Locale.)

commodorejohn
Posts: 1197
Joined: Thu Dec 10, 2009 6:21 pm UTC
Location: Placerville, CA
Contact:

Re: Coding: Fleeting Thoughts

Postby commodorejohn » Sun Aug 11, 2019 10:04 pm UTC

Code: Select all

// universal Unicode handling algorithm
for (int i = 0; i < stringLength; i++) string[i] &= 0x7f;
"'Legacy code' often differs from its suggested alternative by actually working and scaling."
- Bjarne Stroustrup
www.commodorejohn.com - in case you were wondering, which you probably weren't.

User avatar
Link
Posts: 1419
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Mon Aug 12, 2019 8:27 am UTC

Ha!


Return to “Coding”

Who is online

Users browsing this forum: No registered users and 8 guests