User Data Refinement
 |
||
|
User Data Refinement |
|
Perhaps I should have written this first, as a couple of people have pointed out (see the section Motivation below).
A userdata has both a metatable and an environment table. It seems logical that the metatable contains information which is general to the datatype of the userdata, while the environment table contains information which is specific to the instance of the userdata. (See Footnote at bottom. How do you insert anchors into this Wiki?)
The methods also have an environment table. Methods are typically common to all instances of a userdata type; one would expect their environment tables to refer to the userdata's metatable (or some other table shared by all type instances). Unless the methods themselves are specific to the instance, it is hard to imagine a case where the environment table of the method function would be the environment table of the userdata.
So we would normally expect some sort of relationship (possibly equality) between:
(see Code Snippet 1 below.)
The current API for creating a userdata by default initializes the userdata's metatable to NULL and its environment table to the environment table of the caller. But that seems inverted from the common case, which, based on the above analysis, would be to initialize the userdata's metatable to the environment table of the caller, and its environment table to NULL. (Userdata environment tables cannot be NULL in the current implementation, but see below.)
(See Code Snippet 2 below.)
Now, it may be that a given userdata type always requires an environment table (such as Mike Pall's useful example of queues, although even then an empty queue might not require an environment table), but there are also use cases where the environment table is optional. For example, one might use the environment table to store instance properties which belong to the scripting environment rather than to the C implementation, or even to use it to allow userdata method's to be overridden for a particular instance by the Lua code. (This implies an implementation of the __index metamethod which does appropriate lookups.)
The current implementation has no easy way of specifying "no environment table". One could create an empty table, but if the use of the environment table is uncommon in instances of a userdata type, that would be extremely wasteful. The workaround I chose is to just leave the userdata's environment table set to the metatable (i.e. the environment table of the function which created the userdata), and check for that condition. (See Code Snippet 3 below.)
So, in short, userdata environment tables are useful and usable, but the implementation feels awkward because it does not conform to (what I see as) the common case.
About a year ago, I proposed a slightly different (and also flawed) implementation [1]. I now think that proposal was flawed because it tried to recycle the lua_raw* functions, in somewhat the same way that I think the proposed 5.1 implementation is flawed because it tries to recycle the lua_{g,s}etfenv functions. The fact is that the association between a userdata and an instance table is not the same as either a metatable nor an environment table, and the API is more understandable if it does not try to impose an analogy which does not exist.
lua_{g,s}etmetatable() than to lua_{g,s}etfenv(), but which basically have the same effect:
/** * If the indexed object is a metatable and has a peer table, push it onto * the stack and return 1. Otherwise, leave the stack unaltered and return 0 */ int lua_getpeer (lua_State *L, int index); /** * If the indexed object is a metatable, set its peer table to the table * on the top of the stack, or to NULL if the top of the stack is nil, * and return 1, Otherwise return 0. Pop the stack in either case. */ int lua_setpeer (lua_State *L, int index);
The actual code to do this is essentially simply moved from the lua_getfenv() and lua_setfenv() APIs, and does not increase the size of lapi.o by more than a few bytes. The only other modification necessary is in lgc.c where a check must be made for peer being NULL, similar to the check for metatable being NULL.
Also, to cover the common case where metatables are attached to userdata on creation, we augment lua_newuserdata() to take an extra argument, which is the index of the metatable or 0. A common call would be:
self = lua_newuserdatameta(L, sizeof(*self), LUA_ENVIRONINDEX); // but read on
NULL. This change is also trivial.
So far, in terms of the Udata structure, I've proposed little more than a renaming exercise, along with some different creation defaults. The Udata structure has not really been changed, so it continues to suffer from the alignment problem introduced by adding an environment table. In 5.1, the Udata header is now, effectively, five pointer/longs: next, flags, metatable, env, size. If the payload is forced to double-pointer alignment, padding is introduced in the header. If the payload is not forced to double-pointer alignment, it is almost guaranteed to be double-pointer unaligned. (So, for example, in x86 if the payload were a vector of doubles, all of them would be doubleword unaligned.) Consequently, it seems that the cost of adding yet another pointer to the Udata header is fairly small.
In a typical case of a userdata containing a boxed pointer, the payload is only the size of a void*; we could actually put that in the Udata header, and improve alignment (on some platforms, even taking advantage of unused padding.) But in that case, we could consistently set this pointer to the address of the userdata payload, meaning that the payload address could be looked up without a conditional, regardless of whether the userdata was boxed or not. This is very similar to the way UpVals are implemented.
Now, any CFunction which only needs to know the address of the C structure corresponding to the userdata can simply replace lua_touserdata() with lua_tocpeer() and be used with either a boxed or unboxed version of the userdata. In fact, it may be that lua_touserdata() should return the cpeer for full userdata, and that the new API function should be something like lua_topayload().
The metamethod which really cares whether the userdata is boxed or not is the __gc metamethod, if it exists. Fortunately, only two flags are used in CommonHeader, so there is room to insert an isboxed flag byte without bloating Udata's any further. So we just need to add (another!) parameter to the newuserdata API.
Udata *luaS_newudata (lua_State *L, size_t s, Table *e, void *cpeer) {
// ...
u->uv.isboxed = (cpeer != NULL);
u->uv.metatable = e;
u->uv.peer = NULL;
u->uv.cpeer = cpeer ? cpeer : rawuvalue(o) + 1;
// ...
/* One new api function; the other one queries isboxed. */
void *lua_tocpeer (lua_State *L, int index) {
StkId o = index2adr(L, idx);
api_checkvalidindex(L, o);
api_check(L, ttisuserdata(L, o));
return uvalue(o)->cpeer;
}
Finally, some optional niceties for __gc metamethods. Given the above, we might expect a __gc metamethod to look something like this:
int foo_gc (lua_State *L) {
Foo *self = lua_tocpeer(L, 1);
foo_destruct(self); // delete self's references
if (lua_isboxed(L, 1)) foo_free(self); // free self's storage
return 0;
}
Foo object itself would be atomic; that is, without any foo_destruct(). Then it would only be necessary to run the __gc method on a boxed userdata. To facilitate this, we could put two flags in the isboxed byte: LUA_ISBOXED and LUA_NEEDSGC. If the latter flag were off, then the gc would simply delete the object without making any attempt to even look for a __gc metamethod.
2. The datatype-common information typically includes method functions, which would actually be in the table referred to by the __index key in the metatable. Here, I'm assuming the common convention of pointing the metatable __index key back to the metatable itself (possibly mediated by an actual __index function).
-- RiciLake
If it is known that the CFunction environment and the userdata metatable are one and the same, we can use the following instead of luaL_checkudata():
void *luaL_checkself (lua_State *L) {
lua_getmetatable(L, 1);
if (!lua_rawequal(L, -1, LUA_ENVIRONINDEX))
luaL_error(L, "Method called without self or with incorrect self");
lua_pop(L, 1);
return lua_touserdata(L, 1);
}
luaL_checkudata(), this saves a table lookup and a string
comparison; given that this function must be called by every method
(for safety), the time savings can be significant.
The above code could be extended to cover the case where metatable identity is not sufficient to identify the type of a userdata, perhaps because there is more than one applicable metatable. For example, the following would be possible (note that it deliberately leaves the metatable on the stack), and leaves it to the caller to produce an error message):
void *luaL_getselfmeta (lua_State *L) {
lua_getmetatable(L, 1);
if (!lua_isnil(L, -1)) {
lua_pushvalue(L, LUA_ENVIRONINDEX);
lua_gettable(L, -2); // Are we one of the metatable's peers?
if (!lua_isnil(L, -1)) {
lua_pop(L, 1); // Ditch the sentinel. Could have been pop 2
return lua_touserdata(L, 1);
}
}
return NULL;
}
Approximate code for creating packages and userdata themselves. This code is untested; the actual binding system I use is slightly different.
int luaopen_foo (lua_State *L) {
// Check that the typename has not been used
lua_getfield(L, LUA_REGISTRYINDEX, FOO_TYPENAME);
if (!lua_isnil(L, -1))
// Instead of throwing an error, we could just use the returned table
luaL_error(L, LUA_QS "is already in use.", FOO_TYPENAME);
// Make the metatable
lua_newtable(L);
// Register it in the Registry
lua_pushvalue(L, -1);
lua_setfield(L, LUA_REGISTRYINDEX, FOO_TYPENAME);
// Arrange for methods to inherit the metatable as env table
lua_pushvalue(L, -1);
lua_replace(L, LUA_ENVIRONINDEX);
// Fill in the metatable
luaL_openlib(L, NULL, mytypemethod_reg, 0);
// Make the actual package
luaL_openlib(L, MYTYPE_PACKAGE, mytypepkg_reg, 0);
return 1;
}
newobj = lua_newuserdata(L, sizeof(*newobj)); lua_pushvalue(L, LUA_ENVIRONINDEX); lua_setmetatable(L, -2);
newobj = lua_newuserdata(L, sizeof(*newobj));
lua_getfield(L, LUA_REGISTRYINDEX, FOO_TYPENAME);
if (lua_isnil(L, -1))
luaL_error(L, "Userdata type " LUA_QS " has not been registered",
FOO_TYPENAME);
// Set both the metatable and the environment table
lua_pushvalue(L, -1);
lua_setfenv(L, -3);
lua_setmetatable(L, -2);
CFunction's environment table, which we assume it the same as the userdata's metatable (or at least where type methods might be found). The trick is that the userdata's environment table is set to the metatable to indicate that there is no specific environment table; this allows us to save a lookup. However, as can be seen in Comparative Code Snippets (below), we can do even better with a small change to the API.
// Push the value of the indicated field either from the environment
// table of the indexed userdata or from the environment table of the
// calling function.
void getenvfield (lua_State *L, int index, const char *fieldname) {
lua_getfenv(L, index);
lua_getfield(L, -1, fieldname);
if (lua_isnil(L, -1)
&& !lua_rawequal(L, -2, LUA_ENVIRONINDEX)) {
lua_pop(L, 2);
lua_getfield(L, LUA_ENVIRONINDEX, fieldname);
}
else
lua_replace(L, -2);
}
// Put the value on the top of the stack in the environment of the
// indexed userdata with the specified fieldname
void setenvfield (lua_State *L, int index, const char *fieldname) {
lua_getfenv(L, index);
if (lua_rawequal(L, -1, LUA_ENVIRONINDEX)) {
lua_pop(L, 1);
lua_newtable(L);
lua_pushvalue(L, -1);
lua_setfenv(L, index); // Only works if index > 0
}
lua_insert(L, -2);
lua_setfield(L, -2, fieldname);
}
Create a boxed userdata from within a method of the userdata.
void newboxed_self (lua_State *L, void *obj) {
void **newbox = lua_newuserdata(L, sizeof(*newbox));
lua_pushvalue(L, LUA_ENVIRONINDEX);
lua_setmetatable(L, -2);
*newbox = obj;
}
void newboxed_type (lua_State *L, const char *typename, void *obj) {
void *newobj = lua_newuserdata(L, sizeof(*newobj));
lua_getfield(L, LUA_REGISTRYINDEX, FOO_TYPENAME);
if (lua_isnil(L, -1))
luaL_error(L, "Userdata type " LUA_QS " has not been registered",
FOO_TYPENAME);
// Set both the metatable and the environment table
lua_pushvalue(L, -1);
lua_setfenv(L, -3);
lua_setmetatable(L, -2);
}
With peer tables:
void newboxed_self (lua_State *L, void *obj) {
lua_newuserdata_ex(L, 0, LUA_ENVIRONINDEX, obj);
}
void newboxed_type (lua_State *L, const char *typename, void *obj) {
lua_getfield(L, LUA_REGISTRYINDEX, typename);
if (lua_isnil(L, -1))
luaL_error(L, "Userdata type " LUA_QS " has not been registered",
typename);
lua_newuserdata_ex(L, 0, -1, obj);
lua_replace(L, -2);
}
void getenvfield (lua_State *L, int index, const char *fieldname) {
lua_getfenv(L, index);
lua_getfield(L, -1, fieldname);
if (lua_isnil(L, -1)
&& !lua_rawequal(L, -2, LUA_ENVIRONINDEX)) {
lua_pop(L, 2);
lua_getfield(L, LUA_ENVIRONINDEX, fieldname);
}
else
lua_replace(L, -2);
}
void setenvfield (lua_State *L, int index, const char *fieldname) {
lua_getfenv(L, index);
if (lua_rawequal(L, -1, LUA_ENVIRONINDEX)) {
lua_pop(L, 1);
lua_newtable(L);
lua_pushvalue(L, -1);
lua_setfenv(L, index); // Only works if index > 0
}
lua_insert(L, -2);
lua_setfield(L, -2, fieldname);
}
void getpeerfield (lua_State *L, int index, const char *fieldname) {
if (lua_getpeer(L, index)) {
lua_getfield(L, -1, fieldname);
if (!lua_isnil(L, -1)) {
lua_replace(L, -2);
return;
}
}
lua_getfield(L, LUA_ENVIRONINDEX, fieldname);
}
void setpeerfield (lua_State *L, int index, const char *fieldname) {
if (!lua_getpeer(L, index)) {
lua_newtable(L);
lua_pushvalue(L, -1);
lua_setpeer(L, index); // Still only works if index > 0
}
lua_insert(L, -2);
lua_setfield(L, -2, fieldname);
}
index2adr(). The numbers below are api calls/index2adr calls:
current proposed
newself: 3/2 1/1
newtype: 6/5 4/4
getfield (* common case):
peer, found in peer: 4/4 4/4
peer, found in fn env; 6/7 5/5
peer, not found: 6/7 5/5
*No peer, found in fn env: 4/4 2/2
No peer, not found: 5/6 2/2
setfield (* common case):
*peer 4/5 3/3
no peer 8/8 6/5
Here's most of the changes in pseudo-patch format (! indicates a change, + an addition, - a deletion). None of this code has actually been tried :)
/* In lobject.h */
typedef union Udata {
L_Umaxalign dummy; /* ensures maximum alignment for `local' udata */
struct {
CommonHeader;
+ lu_byte isboxed;
struct Table *metatable;
! struct Table *peer;
+ void *cpeer;
size_t len;
} uv;
} Udata;
/* In lstring.c; the header needs to be changed as well */
! Udata *luaS_newudata (lua_State *L, size_t s, Table *e, void *cpeer) {
Udata *u;
if (s > MAX_SIZET - sizeof(Udata))
luaM_toobig(L);
u = cast(Udata *, luaM_malloc(L, s + sizeof(Udata)));
u->uv.marked = luaC_white(G(L)); /* is not finalized */
u->uv.tt = LUA_TUSERDATA;
+ u->uv.isboxed = (cpeer != NULL);
u->uv.len = s;
! u->uv.metatable = e;
! u->uv.peer = NULL;
+ u->uv.cpeer = cpeer ? cpeer : rawuvalue(o) + 1;
/* chain it on udata list (after main thread) */
u->uv.next = G(L)->mainthread->next;
G(L)->mainthread->next = obj2gco(u);
return u;
}
/* in lapi.c */
+ LUA_API void *lua_tocpeer (lua_State *L, int idx) {
+ StkId o = index2adr(L, idx);
+ api_checkvalidindex(L, o);
+ api_check(L, ttisuserdata(L, o));
+ return uvalue(o)->cpeer;
+ }
+ LUA_API int lua_isboxed (lua_State *L, int idx) {
+ StkId o = index2apr(L, idx);
+ api_checkvalidindex(L, o);
+ api_check(L, ttisuserdata(L, o));
+ return uvalue(o)->isboxed;
+ }
! LUA_API void *lua_newuserdata_ex (lua_State *L, size_t size,
! int idx, void *cpeer) {
Udata *u;
+ Table *h = NULL;
lua_lock(L);
luaC_checkGC(L);
+ if (idx) {
+ api_check(L, ttistable(index2adr(L, idx)));
+ h = hvalue(index2adr(L, idx));
+ }
! u = luaS_newudata(L, size, h, cpeer);
setuvalue(L, L->top, u);
api_incr_top(L);
lua_unlock(L);
return u + 1;
}
LUA_API void lua_getfenv (lua_State *L, int idx) {
StkId o;
lua_lock(L);
o = index2adr(L, idx);
api_checkvalidindex(L, o);
! if (ttype(o) == LUA_TFUNCTION) {
- case LUA_TFUNCTION:
sethvalue(L, L->top, clvalue(o)->c.env);
+ }
+ else {
- break;
- case LUA_TUSERDATA:
- sethvalue(L, L->top, uvalue(o)->env);
- break;
- default:
setnilvalue(L->top);
break;
}
api_incr_top(L);
lua_unlock(L);
}
+ LUA_API int lua_getpeer (lua_State *L, int idx) {
+ const TValue *o;
+ Table *peer = NULL;
+ int res;
+ lua_lock(L);
+ o = index2adr(L, idx);
+ api_checkvalidindex(L, o);
+ if (ttype(o) == LUA_TUSERDATA)
+ peer = uvalue(o)->peer;
+ if (peer == NULL)
+ res = 0;
+ else {
+ sethvalue(L, L->top, h);
+ api_incr_top(L);
+ res = 1;
+ }
+ lua_unlock(L);
+ return res;
+ }
LUA_API int lua_setfenv (lua_State *L, int idx) {
StkId o;
int res = 1;
lua_lock(L);
api_checknelems(L, 1);
o = index2adr(L, idx);
api_checkvalidindex(L, o);
api_check(L, ttistable(L->top - 1));
- switch (ttype(o)) {
- case LUA_TFUNCTION:
+ if (ttype(o) == LUA_TFUNCTION) {
clvalue(o)->c.env = hvalue(L->top - 1);
- break;
- case LUA_TUSERDATA:
- uvalue(o)->env = hvalue(L->top - 1);
- break;
- default:
- res = 0;
- break;
- }
luaC_objbarrier(L, gcvalue(o), hvalue(L->top - 1));
+ }
+ else
+ res = 0;
L->top--;
lua_unlock(L);
return res;
}
+ LUA_API int lua_setpeer (lua_State *L, int idx) {
+ TValue *o;
+ Table *peer;
+ int res;
+ lua_lock(L);
+ api_checknelems(L, 1);
+ o = index2adr(L, idx);
+ api_checkvalidindex(L, o);
+ if (ttisnil(L->top - 1))
+ peer = NULL;
+ else {
+ api_check(L, ttistable(L->top - 1));
+ peer = hvalue(L->top - 1);
+ }
+ if (ttype(obj) == LUA_TUSERDATA) {
+ uvalue(obj)->peer = peer;
+ if (peer != NULL)
+ luaC_objbarriert(L, rawuvalue(obj), peer);
+ res = 1;
+ }
+ else
+ res = 0;
+ L->top--;
+ lua_unlock(L);
+ return res;
+ }
/* In lua.h */
+ LUA_API void *lua_tocpeer (lua_State *L, int index);
+ LUA_API int lua_isboxed (lua_State *L, int idx);
+ LUA_API void *lua_newuserdata_ex (lua_State *L, size_t size,
+ int idx, void *cpeer);
! #define lua_newuserdata(L,sz) lua_newuserdata_ex(L, sz, 0, NULL)
+ LUA_API int lua_getpeer (lua_State *L, int idx);
+ LUA_API int lua_setpeer (lua_State *L, int idx);
/* in lgc.c, reallymarkobject */
case LUA_TUSERDATA: {
Table *mt = gco2u(o)->metatable;
+ Table *peer = gco2u(o)->peer;
gray2black(o); /* udata are never gray */
if (mt) markobject(g, mt);
! if (peer) markobject(g, peer);
return;
}
Any table can then be set as the environment that can replace the environment for an object. Yes?
Yes, indeed. But you cannot set "no table" as the environment for an object.
Consider the following case: I bind myFancyWidget to a Lua userdata and export it into the Lua environment.
The Lua script may want to override some method in a particular instance of MyFancyWidget (to make it fancier, maybe :). Now, it could create an entire new object to do that, but it would be a lot simpler to do this:
function overrideCtlY(widget)
local oldDoKeyPress = myFancyWidget.doKeyPress
function widget:doKeyPress(key)
if key == "ctl-y" then
-- handle control y the way I want to
else
return oldDoKeyPress(widget, key)
end
end
return widget
end
local widget = overrideCtlY(MyFancyWidget.new())
doKeyPress member function. I can't store it in the standard metatable; that would apply to all instances. Logically, I should store it in the widget's environment table, since that is local to the widget instance.
In the common case, of course, no methods are overridden. So I don't want an environment table at all; I want the method lookup to go directly to the metatable. If I can't set the environment table to nil, then I have to set it to some sentinel and test for that on every lookup. So I was looking for something that:
a) semantically corresponded to (my) expected use of environment tables.
b) involved fewer API calls on common operations.
The goal, then, is not profound. It simply reflects my thought that setting the env table of a userdata to the env table of the currently running function is an extremely unlikely default, and that being able to set it to nil is a useful feature.