MagicServer: a fake server for testing

[samuelcolvin]

Like MagicMock but for testing requests:

• Logs requests for later check by test
• Generally returns 200 response
• If URL includes 404, return 404 etc.
• Equivalent of return_value and side_effect
• Use accept/content-type to decide response format
• Available as pytest fixture or for unittest

Thoughts/ideas?

[kxepal]

What this server will test if it suppose to always respond with 200?

If you want fake unit testing e.g. just client side, you should mock the request calls, setting the deserved response. If you want to do integrational testing, you need to test against real server instance.

[asvetlov]

@kxepal let's assume I should test my system which integrates with facebook or github.
Fake facebook server looks reasonable for the case.

[kxepal]

@asvetlov
Fake Facebook server you can setup with aiohttp.web.Application - just implement endpoints you interested in, simulate original server behavior and run in background. But unlikely you would need in it - too much to code when result is still unknown.

You can just simply mock all requests responses, defining what more-or-less Facebook will return to you and how your client would react on that. Same result, but less to code and no network round trips (faster tests in other words). Moreother, this way will allow you to make tests against exact real Facebook responces and track down regressions. With fake server I'm not sure how you'll do the same.

[jettify]

There is https://github.com/pnuckowski/aioresponses which is clone of https://github.com/getsentry/responses but for aiohttp.

@aioresponses()
def test_http_headers(m):
    loop = asyncio.get_event_loop()
    session = aiohttp.ClientSession()
    m.post( 'http://facebok.com', payload=dict(), headers=heders)

    resp = loop.run_until_complete(session.post('http://example.com'))

Agree with @kxepal , mocking looks like better approach.

Side note: I implemented fake server few times to test download and retry logic. Basically I had aiohttp server and simulated different failures:

1. bad status code
2. timeouts (connect, read)
3. server drops connections after few bytes send
4. content ranges
5. wrong content-types
   and so on.

[argaen]

+1 for mocks over fake server. Doing a fake server you have to keep maintaining it as a new piece of software, services you connect to keep evolving and implementing all cases like timeouts, bad responses, different status is hard with fake servers while with mocks usually is couple of lines

[samuelcolvin]

Doing a fake server you have to keep maintaining it as a new piece of software

Quite the reverse, aioresponse is a separate piece of software which is never going to exactly match a real response. Generally running a real aiohttp server guarantees realistic responses. Because you're communicating with a real (if very close) server.

Personally I find creating and using a real server very quick and haven't had issues at all with maintaining them ever as aiohttp has developed.

The whole point in MagicServer is that it would make using a server to test requests a one-liner.

[kxepal]

@samuelcolvin

Personally I find creating and using a real server very quick and haven't had issues at all with maintaining them ever as aiohttp has developed.

The whole point in MagicServer is that it would make using a server to test requests a one-liner.

Could you please show (as concept) how it would, because these two statements looks like conflicting for me (fake server vs mock one).

[argaen]

The whole point in MagicServer is that it would make using a server to test requests a one-liner

I achieve the same with mocks (I don't use aioresponses, just asynctest). Anyway as @kxepal comments maybe a practical example would help to understand

[fafhrd91]

MagicServer works as specification, as spec evolves server impl changes. then it is easy to make sure software is compatible with spec. Mock solves different pproblem. I think, excessive mock usage is bad idea.

[fafhrd91]

Good example is sockjs library, it has sockjs-protocol.py tests, so it is very easy to create server implementation

[samuelcolvin]

Far from perfect, but hopefully this demonstrates how it would work:

import pytest
import aiohttp
from aiohttp import web


@web.middleware
async def logging_middleware(request, handler):
    try:
        r = await handler(request)
    except web.HTTPException as e:
        request.app['request_log'].append(f'{request.method} {request.path_qs} > {e.status}')
        raise
    except Exception:
        request.app['request_log'].append(f'{request.method} {request.path_qs} > 500')
        raise
    else:
        request.app['request_log'].append(f'{request.method} {request.path_qs} > {r.status}')
        return r


def create_magic_app():
    app = web.Application(middlewares=[logging_middleware])
    app['request_log'] = []
    return app


@pytest.fixture
def magic_server(loop, test_server):
    app = create_magic_app()
    server = loop.run_until_complete(test_server(app))
    server.url = f'http://localhost:{server.port}'
    return server


async def foobar(endpoint):
    """
    This is the function to test
    """
    async with aiohttp.ClientSession() as session:
        async with session.get(endpoint) as r:
            t = await r.text()
        if 'hello' in t:
            async with session.get(endpoint + 'whatever/') as r:
                if r.status == 409:
                    return 'foo'
                else:
                    return 'bar'


async def test_index(request):
    ...
    return web.Response(text='hello')


async def test_foobar(magic_server):
    # FIXME: the following won't work yet, would need someway to freeze the app on first request
    # or add endpoints after freezing

    # add_get, add_post, add_route work like app.router.* methods, but allow responses as arguments
    magic_server.add_get('/', test_index)  # will call test_index and return result
    magic_server.add_post('/whatever/', web.HTTPConflict())  # will just return 409

    v = await foobar(server.url)
    assert v == 'foo'
    assert magic_server.app['request_log'] == [
        'GET / > 200',
        'POST /whatever/ > 409',
    ]

[samuelcolvin]

A few notes:

• unlike my initial suggestion, I think it should return 404 if endpoint isn't implemented (this is unlike MagicMock which tries say "yes" to everything, but more sensible I think)
• obviously more complex failures like invalid http respones are possible I think
• helper methods like requested('GET', '/') -> bool and request_count('POST', '.*') -> int would be useful.

[argaen]

I still feel mocking is way simpler

with patch('my_module.ClientSession.get', return_value=ResponseLikeObject):
    await do_your_request()

Also faster since there is no need to setup and start app for every test and even if you would make the server fixture session scoped then you would have risks of being stateful during the tests run.

[samuelcolvin]

I run thousands of tests with test scoped server, the time taken is tiny.

…

On 13 Nov 2017 22:55, "Manuel Miranda" ***@***.***> wrote: I still feel mocking is way simpler with patch('my_module.ClientSession.get', return_value=ResponseLikeObject): await do_your_request() Also faster since there is no need to setup and start app for every test and even if you would make the server fixture session scoped then you would have risks of being stateful during the tests run. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2512 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD2jGU3NPJMU5nBC8a8YyufoYupbOZDRks5s2MjagaJpZM4Qa3gZ> .

[argaen]

In that case I guess its more of a personal opinion of what to use. Its not like its going to force anyone to use it so no strong opinion on whether it makes to officially to aiohttp or not

[ambivalentno]

Sorry for intrusion, but, given I've spent some time converting https://github.com/aio-libs/aiohttp/blob/master/examples/fake_server.py to tests, I'll leave a gist for newbies here https://gist.github.com/ambivalentno/e311ea008d05938ac5dd3048ce76e3d1

Also, I have to note that if you've got 10 apis with similar logic, but different urls/ response schemas, it's much easier to write an abstract test server to encapsulate their logic and next add routes and serialization only. imho.